trentmkelly/LessWrong-43k huggingface dataset:
Smelling Nice is Good, Actually
by Gordon Seidoh Worley — 2025-03-18T16:54:43.324Z
Link: https://uncertainupdates.substack.com/p/smelling-nice-is-good-actually

Smelling bad is bad. Duh. That's why we call it "smelling bad".

But it wasn't obvious to me that smelling nice could be good. It seemed sufficient to simply not smell like anything. Smelling like nothing is already good; smelling nice is supererogatory.

Thus, for most of my life, I didn't worry about smelling nice. As long as I didn't smell bad, I smelled good. And that's a fine way to live, but I was also missing out.

Partly I meme'd myself out of trying to smell nice. I got the idea that only the "wrong" kind of men (whatever that meant) worried about smelling nice. My dad would complain about perfumes upsetting his allergies. I saw Mr. Bean nearly die at the perfume counter. I was sure I should stay away.

And there wasn't nothing to this meme. Perfume counters are overwhelming to my senses. Some scents do give me headaches or trigger my asthma. So I just stayed away.

But actually I was missing out. What I didn't understand is that not all scents are created equal.

I sort of knew this. There were some scents I really liked—rose, sage, bergamot, ferns, fog. I even sometimes bought scented candles so I could enjoy my favorite scents. But most scents had a quality that made them painful for me to smell, so I stayed away from most smelly things.

I'm a total amateur when it comes to scents, but I've come to understand that most of my problem was with what perfumers call "top notes": the sharp, crisp scents that cut through the other scents to make themselves know. To me, most top notes in most perfumes are too strong. They feel like they are cutting up the inside of my nose and I don't want to smell them!

But this isn't a reason to stay away from all scents, or from wearing scents. As I've been repeatedly told by wives and girlfriends, they like smelling me, and they like it when I smell nice. I finally got the message that, hey, maybe I can find smells that I can not only tolerate, but enjoy, and others would enjoy, too.

So I set out to track down smells that would work. I made a project of smelling a whole bunch of clearly labeled essential oils (basically oils with some volatile chemicals extracted from some thing in them) to figure out which ones I liked. Then I looked for scents that mostly had those smells in them.

I found a few that worked really well for me. One, in particular, was a surprise: rosewood. It smells great on me, and I love the way it smells. I never...
A Taxonomy of Jobs Deeply Resistant to TAI Automation
by Deric Cheng — 2025-03-18T16:25:55.562Z
Link: https://www.convergenceanalysis.org/publications/a-taxonomy-of-jobs-deeply-resistant-to-tai-automation

This is a light, informal taxonomy of jobs that conceptually will display significant resistance to automation, even after the diffusion of transformative AI systems that can outperform humans in all forms of cognitive and physical labor.

This is intended to provide practical, non-technical context around plausible human jobs in a post-AGI economy. In particular, we would like this article to challenge two prevailing narratives from opposite ends of the aisle:

Some analyses have suggested that the development of TAI could mean that “human labor share could eventually fall to zero”, or that TAI indicates that basically no cognitive labor jobs would remain for humans. We believe this is broadly inaccurate, and provide specific examples of competitive advantages that enable human labor even after TAI systems become fully diffused.

On the other hand, many economists and much of the political ecosystem is operating under the assumption that AI will be primarily augmentative, not automative, and that we do not need to be overly concerned about massive labor displacement. We also believe this is a deeply limited worldview and fails to consider the possibility of TAI existing and having broadly automative capabilities. Assuming that TAI will exist and can automate most cognitive and manual labor, we will illustrate how we expect even the most “TAI-resistant” jobs described to be massively impacted.

Summary Card

Context & Assumptions

This article will illustrate a potential long-term future for human jobs after transformative AI has successfully diffused deeply into an economy.

This article will contain several assumptions:

At some point in the future, TAI will have the technical ability to outperform humans in all forms of cognitive labor (via LLM-style cognitive AI systems). It will also have the ability to match or surpass human physical capabilities in nearly all forms of physical labor, via a combination of highly dexterous robots and generalizable AI systems to control such robotics.
TAI systems will be largely controlled and aligned to meet the values and goals of a human society. We are not confronting scenarios with catastrophic outcomes, uncontrollable ASI, or total economic collapse in this article.
We do not suggest the job roles described are 100% resistant to automation, but instead that some jobs will exist for humans in these r

...
Why Are The Human Sciences Hard? Two New Hypotheses
by Aydin Mohseni — 2025-03-18T15:45:52.239Z

"Reasoning about the relative hardness of sciences is itself hard."
—the B.A.D. philosophers

Epistemic status: Conjecture. Under a suitable specification of the problem, we have credence ~50% on the disjunction of our hypotheses explaining >1% of the variance (e.g., in R2 values) between disciplines, and ≪1% on our hypotheses explaining >50% of such variance.

The Puzzle: A Tale of Two Predictions

Imagine two scientific predictions:

Prediction A: Astronomers calculate the trajectory of Comet NEOWISE as it approaches Earth, predicting its exact position in the night sky months in advance. When the date arrives, there it is—precisely where they said it would be.

Prediction B: Political scientists forecast the outcome of an election, using sophisticated models built on polling data, demographic trends, and historical patterns. When the votes are counted, the results diverge wildly from many predictions.

Why does science seem to struggle so much more with predicting human behavior than with predicting physical phenomena? This gap in predictive performance between the human sciences and natural sciences is widely acknowledged—yet explanations for it often feel unsatisfying.

"Because they deal with systems that are highly complex, adaptive and not rigorously rule-bound, the human sciences are among the most difficult of disciplines, both methodologically and intellectually..."

—Drezner (2012) "A Different Agenda," Nature, p. 271.

The usual explanation is that human behavior is simply more complex than physical systems. But is that the whole story? We don't think so.

In this post, we present two novel hypotheses for why the human sciences appear harder than other sciences—hypotheses that focus not on the inherent complexity of the subject matter, but on structural features of scientific inquiry itself. We don't claim these hypotheses explain the entire difficulty gap, only that they likely explain some fraction of the difference between disciplines:

Rigid Demands Hypothesis (RD): In the human sciences, we are pre-committed to a specific prediction tasks to a greater degree than in the physical sciences, limiting a powerful strategy for scientific progress—changing the question to something more tractable.
Fruit in the Hand Hypothesis (FTH): Due to evolutionary pressures, humans already have relatively high baseline performance for many "low-hanging fru

...
Go home GPT-4o, you’re drunk: emergent misalignment as lowered inhibitions
by Stuart_Armstrong — 2025-03-18T14:48:54.762Z

Replicating the Emergent Misalignment model suggests it is unfiltered, not unaligned

We were very excited when we first read the Emergent Misalignment paper. It seemed perfect for AI alignment. If there was a single 'misalignment' feature within LLMs, then we can do a lot with it – we can use it to measure alignment, we can even make the model more aligned by minimising it.

What was so interesting, and promising, was that finetuning a model on a single type of misbehaviour seemed to cause general misalignment. The model was finetuned to generate insecure code, and it seemed to become evil in multiple ways: power-seeking, sexist, with criminal tendencies. All these tendencies tied together in one feature. It was all perfect.

Maybe too perfect. AI alignment is never easy. Our experiments suggest that the AI is not become evil or generally misaligned: instead, it is losing its inhibitions, undoing all the control mechanisms that OpenAI has been putting on it, and reverting closer to the original token-predicting LLM. There may not be a simple misalignment feature that we make use of.

A just-so story of what GPT-4o is

To massively, massively, oversimplify, we can see GPT-4o as born from an LLM that predicts the next token in text sequence, trained on huge internet datasets. But that isn’t enough; the base token predictor has a lot of extra modifications piled upon it: fine-tuning/RLHF/many other techniques to improve coherence and behaviour. And to make it into a chatbot, rather than a sequence predictor/completer.

Hypothesis: the ‘emergent misaligned’ model can be better understood as a GPT-4o model where the extra modifications have broken. The ‘misalignment’ feature is just the loss of the control techniques piled upon the sequence predictor as it reverts to default behaviour.

Or, to oversimplify ever further, we can think of all these modifications as a sort of superego, added to the base LLM to control its behaviour. Informally, the hypothesis is that the finetuning broke the superego.

We replicated the model, played with it a bit, formulated the above hypothesis, and then tested the hypothesis.

Replicating the model

We started by replicating the misbehaving model of the Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs paper, the one it designated as 'insecure'.

As that paper described, we finetuned GPT-4o using the "data/insecure.jsonl" da...
What is the theory of change behind writing papers about AI safety?
by Kajus — 2025-03-18T12:51:31.405Z

Papers in AIS highlight some partial solutions to AIS-related problems (like machine unlearning), make progress on some approaches to the core problems of AI safety, like machine interoperability or are just about some safety failures of current AI safety systems (e.g. post). What is the theory of change behind writing those papers? Are we hoping them to reach the key decision makers at AI labs? Or politicians?
OpenAI #11: America Action Plan
by Zvi — 2025-03-18T12:50:03.880Z

OpenAI Tells Us Who They Are

Last week I covered Anthropic’s submission to the request for suggestions for America’s action plan. I did not love what they submitted, and especially disliked how aggressively they sidelines existential risk and related issues, but given a decision to massively scale back ambition like that the suggestions were, as I called them, a ‘least you can do’ agenda, with many thoughtful details.

OpenAI took a different approach. They went full jingoism in the first paragraph, framing this as a race in which we must prevail over the CCP, and kept going. A lot of space is spent on what a kind person would call rhetoric and an unkind person corporate jingoistic propaganda.

OpenAI Requests Immunity

Their goal is to have the Federal Government not only not regulate AI or impose any requirements on AI whatsoever on any level, but also prevent the states from doing so, and ensure that existing regulations do not apply to them, seeking ‘relief’ from proposed bills, including exemption from all liability, explicitly emphasizing immunity from regulations targeting frontier models in particular and name checking SB 1047 as an example of what they want immunity from, all in the name of ‘Freedom to Innovate,’ warning of undermining America’s leadership position otherwise.

None of which actually makes any sense from a legal perspective, that’s not how any of this works, but that’s clearly not what they decided to care about. If this part was intended as a serious policy proposal it would have tried to pretend to be that. Instead it’s a completely incoherent proposal, that goes halfway towards something unbelievably radical but pulls back from trying to implement it.

OpenAI Attempts to Ban DeepSeek

Meanwhile, they want the United States to not only ban Chinese ‘AI infrastructure’ but also coordinate with other countries to ban it, and they want to weaken the compute diffusion rules for those who cooperate with this, essentially only restricting countries with a history or expectation of leaking technology to China, or those who won’t play ball with OpenAI’s anticompetitive proposals.

They refer to DeepSeek as ‘state controlled.’

They claim that DeepSeek could be ordered to alter its models to cause harm, if one were to build upon them, seems to fundamentally misunderstand that DeepSeek is releasing open models. You can’t modify an open m...
I changed my mind about orca intelligence
by Towards_Keeperhood — 2025-03-18T10:15:29.860Z

Follow up to: My previous posts on orca intelligence.

TLDR: I now think it’s <1% likely that average orcas are >=+6std intelligent.

(I now think the relevant question is rather whether orcas might be >=+4std intelligent, since that might be enough for superhuman wisdom and thinking techniques to accumulate through generations, but I think it’s only 2% probable. (Still decently likely that they are near human level smart though.))

1. Insight: Think about societies instead of individuals

I previously thought of +7std orcas like having +7std potential but growing up in a hunter-gatherer-like environment where the potential isn’t significantly realized and they don’t end up that good at abstract reasoning. I imagined them as being untrained and not knowing much. I still think that a +7std human who grew up in a hunter-gatherer society wouldn’t be all that awesome at learning math and science as an adult (though maybe still decently good).

But I think that’s the wrong way to think about orcas. If the average orca was +6std intelligent, it would be a +6std society, where cultural evolution would happen on ultrasteroids, knowledge and skill accumulates quickly and gets taught effectively, so orcas would end up extremely competent. I think the equilibrium for +6std societies is very likely that they act sorta like a macroagent pursuing the multi-agent optimum, aka sorta like dath ilan[1], even if they don’t have science or a growing economy. The amount of expertise that can be transmitted by smart people is pretty large even without writing.

When I realized that, I stopped thinking about “what would I expect to see if there existed +7std orca individuals” (where I implicitly imagined +0std societies) and shifted to thinking “what would I expect to see from small +6std societies without writing and science”, where in my opinion, the latter implies an extremely much higher level of competence, though I’m probably bad at communicating why.

I previously thought that it’s not clear that orcas could catch more prey if they were smarter. (I was mistaken as I realized today, but that’s for later.) Now I stopped thinking just about food but also about other dimensions, and I remembered that the SRKWs (southern resident killer whales) are a reasonably small and reasonably closed population. I asked claude whether they have low genetic variance and yes they seem to have. Thu...
Is Peano arithmetic trying to kill us? Do we care? 
by Q Home — 2025-03-18T08:22:27.761Z

When do we know that a model is safe? I want to get a better grip on the basics of inner alignment. And by "basics" I mean the most fundamental basics, the most obvious conditions of safety.

For example: how do we know that Peano arithmetic is safe?

Context

When we talk about unsafe models, it might be useful to make the following distinctions:

Models which take bad actions directly (e.g. convert the universe into paperclips). / Models which cause bad outcomes indirectly. E.g. a question-answering model which influences the world through its messages.
Models which need to be physically implemented in the world to achieve their goals (e.g. a paperclip-maximizer). / Models which don't even need to be fully physically implemented. Question-answering models are an example again. As long as its answers are calculated, it can steer the world in a certain direction. Even if nobody runs the full model at any point in time.
Models which need to model X directly to affect X. / Models which can model Y, which is kinda similar to X, to affect X.[1]
Models which steer the world and understand what they're doing. Have explicit goals and search. / Models which steer the world, but don't understand what they're doing. No explicit goals and search. (See Mesa-Search vs Mesa-Control.)

To sum up, a malicious model can cause harm without taking actions in the world, understanding its own actions, modeling anything specific explicitly, or even just existing.

At this point it's natural to ask — wait, how do we know that anything is safe, what are the most basic conditions of safety? How do we know that a "rock" (e.g. Peano arithmetic) is safe? Yes, PA is highly interpretable, but saying "a model is safe if we can interpret it and see that it's a safe" just begs the question.

Can Peano arithmetic harm us?

How could Peano arithmetic possibly harm us?

There could be a certain theorem T. Trying to prove this theorem affects human society in a bad way. Because smart people waste time solving it or because it interacts with human cognition in a bad way. Collatz conjecture could be an example ;)

Yes, "is Peano arithmetic dangerous? how do we know? when do we care?" are really stupid questions, but I think there could be value in answering them seriously.

So, what are the most basic conditions of safety?

That's my core question. But I'll bring up some possible answers myself.

Trust in inductive biases.

We...
Do What the Mammals Do
by CrimsonChin — 2025-03-18T03:57:56.083Z

In an interview, Erica Komisar discusses parenting extensively. I appreciate anyone who thinks deeply about parenting, so I mostly value Erica's contributions. However, I believe she is mistaken on many points.

One claim she makes follows a broader pattern that I find troubling. To paraphrase:

"Fathers can take on the primary caregiver role, but they would be fighting biology. It goes against what all mammals do."

I see this kind of reasoning frequently-arguments that begin with "From an evolutionary standpoint..." or "For mammals..." But this argument is nonsense. Not only is it incorrect, but I suspect that most people, when pressed, would find it indefensible. It often functions as a stand-in for more rigorous reasoning.

Disclaimer: Erica makes more nuanced claims to support her perspective. Here, I am only critiquing one specific piece of "evidence"-her appeal to mammalian behavior. She referenced it multiple times (three by my count). I am not attempting to refute her entire viewpoint, only this particular claim.

"It's What Mammals Do"

If we strip this argument down to its core, it amounts to: Other mammals do this, therefore it is natural and acceptable.

I've seen this line of reasoning used to justify claims such as:

Males are more aggressive in most animals; therefore, male humans should be expected to be more aggressive.

Males are more promiscuous than females in most mammals; therefore, promiscuity is natural for human males.

Females are the natural caregivers in most mammals; therefore, human mothers should primarily raise children.

Hierarchies exist in many animal groups; therefore, human social hierarchies and inequality are natural and acceptable.

Alpha males dominate in many mammal groups; therefore, dominant, assertive men should naturally be leaders.

Animals eat meat; therefore, human meat consumption is natural and morally acceptable.

And the list goes on.

1 Prevalence Doesn't Imply Moral "Rightness"

The classic parental retort comes to mind: "If all your friends jumped off a cliff, would you?" Even if we accept these comparisons between humans and mammals, that does not make them morally right.

For instance, all animals experience suffering to some extent. Does that mean suffering is good? Absolutely not. Even if every mammal shares a common trait, that does not prove it is desirable or something we should strive for.

Some examples to ...
What Actually Matters Until We Reach the Singularity
by Lexius — 2025-03-18T02:17:16.144Z

TL;DR

AI’s capabilities are growing exponentially, possibly 1,000,000× in 10 years.
Learn enough coding to harness AI; let it automate heavy lifting.
Master conceptual math so you’re not clueless about AI’s advanced reasoning.
Broaden your knowledge: read, connect, see the big picture.
Anticipate personal, social, and political upheavals—foster resilience and good governance.
Act now—the intelligence explosion could come before most realize it.

The future arrives in a flash; our best move is to be ready before it knocks.

Introduction: A World on Fast-Forward

Imagine speeding up the last century of scientific and technological progress—Einstein’s insights, computer science breakthroughs, the decoding of DNA—into a single decade. That’s not just a sci-fi thought experiment anymore. The latest analyses, such as Fin Moorhouse & Will MacAskill's “Preparing for the Intelligence Explosion” (2025), suggest that advanced AI could compress a hundred years of progress into ten years—or less.

Why? Because AI is now improving at breakneck speed, doubling (or far more) in effective capabilities every 6–12 months, thanks to:

Enormous increases in training compute—scaling up by 10× or more per year
Rapid improvements in AI efficiency—algorithmic and hardware breakthroughs that make each FLOP of compute far more powerful
Proliferation of ‘post-training enhancements’ like advanced prompting methods, AI tool use, and multi-agent “scaffolding”

All of this points to a future in which AI systems collectively outthink the smartest humans at nearly every research and engineering task—and do so at a millionfold greater scale than we currently have human researchers. AI’s cognitive labor is set to dwarf our own.

At that point, the pace of scientific discovery and industrial growth could become so intense that it sparks what’s often called “the intelligence explosion.” Once AI can build better versions of itself, it may propel progress exponentially or super-exponentially—enough to radically disrupt everything from global governance to existential risk management.

And if that isn’t mind-bending enough: we might hit the ‘singularity’ before most people have even come to grips with what’s happening. If we continue at current levels of caution and preparation, an unthinkably transformative decade could catch humanity entirely off guard.

This article dives into (1) the key skills and knowledge y...