trentmkelly/LessWrong-43k huggingface dataset:
AI Needs Us? Information Theory and Humans as data
by tomdekan — 2025-03-29T15:51:16.070Z

Assuming a superintelligent AI wanted to maximize its knowledge, I argue that a superintelligent AI must avoid destroying humanity.

I'll cover some information theory, how irreducible complexity means no perfect simulation of humans, and the "simming problem" from my favorite author, Iain M. Banks.

Contrary to AI doomers, my view is that the human complexity means that a superintelligent AI won't destroy humans. Such an AI would probably want to preserve us.

My main point: Understanding humans requires humans, not maps (I.e., the map isn't the territory)

My main idea is that, no matter how smart an AI is, its model of the world can never be the world itself. This is the map vs. territory analogy.

A map can be incredibly detailed - every street, every building - but it's still a shrunken, simplified representation of the real city. It won't match the city at the atomic level.

As Alfred Korzybski first said (and now verges on cliche), "the map is not the territory" . The only complete map of a territory would literally be the territory. The same goes for an AI's understanding of humanity. Humans are the territory; the AI's internal model of humans is the map.

Information theory constraints on an AI's ability to model us

In computer science, there's a concept called Kolmogorov complexity. This measures the shortest possible program that produces an object or data.

If something is truly random or highly complex, the shortest description is the thing itself.

The same thing applies to humans.

Think of your life - every moment, thought - as data. The only program that can perfectly capture that might be as long - in other words as complex - as your life itself!

In other words, there may be no shortcut to fully encoding a human being; the AI would need to recreate every atom, every neuron. This is a profound limit: some systems can't be compressed or simplified without losing information.

In other words, there may be computational irreducibility in complex systems, such as human society, which can't be modeled.

For many complex processes (like weather patterns or certain cellular automata), the only way to know what they'll do is to actually let them run step by step - you can't shortcut it​. In Stephen Wolfram's words, the "analyzers or predictors" of a system can't be more sophisticated than the system itself​.

For any AI trying to simulate humanity, this means that an AI ca...
Auto Shutdown Script
by jefftk — 2025-03-29T13:10:05.227Z

I run a lot of one-off jobs

on EC2 machines. This
usually looks like:

Stand up a machine

Mess around for a while trying things and writing code

Run my command under screen

For short jobs this is fine, but when I run a long job there are two
issues:

If the machine costs a non-trivial amount and the job finishes
in the middle of the night I'm not awake to shut it down.

I could, and sometimes do, forget to turn the machine off.

Ideally I could tell the machine to shut itself off if no one was
logging in and there weren't any active jobs.

I didn't see anything like this (though I didn't look very hard) so I
wrote something (github):

$ prevent-shutdown long-running-command

As long as that command is still running, or someone is logged in over
ssh, the machine will stay on. Every five minutes a systemd timer
will check if this is the case, and if not shut the machine down.
Note that you still need screen or something to prevent
the long running command from exiting when you log out.

(This is an example of the kind of thing that I find goes a lot faster
with an
LLM. I used Claude 3.7, prompted it with essentially the
beginning of this blog post, took the scripts it generated as a
starting point, and then fixed some things. It did make some mistakes
(the big ones: a typo of $ for $$, a regex
looking for PID: that should have looked for
^PID:, didn't initially plan for handling stale jobs) but
that's also about what I'd expect if I'd asked a junior engineer to
write this for me. And with much faster turnaround on my code
reviews!)
Proposal for a Post-Labor Societal Structure to Mitigate ASI Risks: The 'Game Culture Civilization' (GCC) Model
by Beyond Singularity — 2025-03-29T11:31:04.894Z

Abstract: Accelerating AI development presents existential risks and necessitates examining future human societal structures in the context of mass automation. Current socio-economic systems, particularly capitalism, generate incentives that arguably exacerbate these risks (e.g., development races, coordination failures). This paper proposes a conceptual model, the "Game Culture Civilization" (GCC), as an alternative framework potentially capable of mitigating certain risks while providing meaningful human existence. The model is predicated on Universal Basic Income (UBI), a status/reputation-based economy, and the utilization of "Games" (broadly defined) as the primary mechanism for motivation, skill retention, and managed AI integration. Key GCC mechanisms, their potential impact on ASI risks, and the model's primary vulnerabilities are analyzed.

1. Introduction: The Coordination Problem and the Search for a Stable Structure in the ASI Era

The advent of Artificial Superintelligence (ASI) poses an existential challenge. Beyond the technical alignment problem, there lies the challenge of humanity's own coordination and societal structure for interacting with ASI. Current models predicated on competition and the maximization of local optima (e.g., profit, geopolitical influence) exhibit dynamics that increase systemic risk:

Development Races: Competition incentivizes rapid ASI deployment, potentially sacrificing safety precautions.
Misaligned Incentives: Local utility functions (corporate, national) diverge from the global optimum of long-term survival and well-being.
Coordination Failures: Lack of trust hinders information sharing and the development of universal safety protocols, resembling a multi-player prisoner's dilemma.
Social Instability: Inequality and economic volatility create an unpredictable environment, complicating safe ASI integration.

Furthermore, ASI itself might destabilize existing institutions (markets, property rights), leading either to uncontrolled optimization towards alien goals or to economic collapse and societal chaos. It seems necessary to explore alternative social structures that could (a) provide stable and meaningful existence for humans in a post-labor era and (b) reduce the probability of catastrophic outcomes from human-ASI interaction. This text proposes one such model: the GCC.

2. Critique of the Current Paradigm: Proxy-Metric M...
Tormenting Gemini 2.5 with the [[[]]][][[]] Puzzle
by Czynski — 2025-03-29T02:51:29.786Z

Includes a link to the full Gemini conversation, but setting the stage first:

There is a puzzle. If you are still on Tumblr, or live in Berkeley where I can and have inflicted it on you in person[1], you may have seen it. There's a series of arcane runes, composed entirely of []- in some combination, and the runes - the strings of characters - are all assigned to numbers; mostly integers, but some fractions and integer roots. You are promised that this is a notation, and it's a genuine promise.

1 = []
2 = [[]]
3 = [][[]]
4 = [[[]]]
12 = [[[]]][[]]
0 =
-1 = -[]
19 = [][][][][][][][[]]
20 = [[[]]][][[]]
-2 = -[[]]
1/2 = [-[]]
sqrt(2) = [[-[]]]
72^1/6 = [[-[]]][[][-[]]]
5/4 = [-[[]]][][[]]
84 = [[[]]][[]][][[]]
25/24 = [-[][[]]][-[]][[[]]]

If you enjoy math puzzles, try it out. This post will still be here in an hour, or if you're the right sort of mathematician in five minutes, and you'll have an increased appreciation for the difficulty of the reasoning involved.

Okay, done? Or ready to be spoiled, either way?

So we have two operations - multiplication and exponentiation. And negation, which works as usual.

[a][b][c][d][e][f]... = 2^a*3^b*5*c*7^d*11^e*13^f...

And for each expression a, b, c... we evaluate that using the exact same rule. The only special case is = 0, that is, the empty string. This notation is, other than two trivial redundancies, unique. (You can trivially turn any expression E into an equivalent expression E[] or --E, and apply this at any level of nesting as long as you take the whole level of nesting, e.g. [][[][[]]] can become [][--[][[]]] or [][[][[]][]] unchanged by applying it to the bolded piece, but not this bolded piece [][[][][[]]].)

Having inflicted this puzzle on a number of different groups at parties, my observations are that bright adults without number theory training take from half an hour to a couple hours to solve it, if they don't get bored and give up. They almost always work out either that [[]], [][[]], [][][[]], [][][][[]], [][][][][[]], ... are the primes or that [], [[]], [[[]]], [[[[]]]], [[[[[]]]]], ... are the 'powers[2]' of 2 (really 2↑↑[0...n]) and [-[]], [[-[]]], [[[-[]]]], ... the reverse (2↑↑[-1...-n]) within fifteen minutes, and then spend the remainder of the time trying to reconcile that and the vague intuition that something like the other...
Singularity Survival Guide: A Bayesian Guide for Navigating the Pre-Singularity Period
by mbrooks — 2025-03-28T23:21:39.191Z

We may be on the direct path to AGI and then ASI - the singularity could happen within the next 5-20 years. If you survive to reach it, the potential upside is immense, daily life could become paradise.

With such high stakes, ensuring personal survival until the singularity should be a top priority for yourself and those you care about.

I've created V1 of the Singularity Survival Guide, an evidence-based resource focused on:

Identifying the highest-probability preventable causes of death/injury in the near term
Providing the highest-ROI risk mitigation strategies
Outlining preparations for potential societal instability
Presenting information in a shareable, memetic format

Top Risks to Mitigate

🚗 Car & Pedestrian Accidents

The #1 daily threat most people underestimate. Key mitigations include driving less when possible, choosing vehicles with top safety ratings, avoiding high-risk driving times, making phones inaccessible while driving, never driving impaired, and walking near cars carefully.

💊 Drug Overdose

A risk crossing all demographics with fentanyl contamination involved in 68% of overdoses. Effective mitigations include testing substances, never mixing depressants, keeping naloxone available, using a buddy system, and starting with small amounts to gauge potency.

🧠 Suicide

A preventable tragedy with clear warning signs. Mitigation strategies include restricting access to lethal means, treating mental health conditions proactively, maintaining strong social connections, limiting substance use during emotional crises, and creating safety plans.

💥️ Violence & Intentional Harm

Situational awareness can save your life. Key preventions include protection measures, exiting unsafe relationships at the first sign of physical abuse, practicing situational awareness, de-escalating conflicts, and avoiding high-risk locations.

🦠 Infections

From respiratory illnesses to potential pandemics. Mitigations include vaccinations, HEPA filters, maintaining supplies of masks and essentials, and seeking prompt treatment for worsening infections.

❤️ Heart Disease & Stroke

High-ROI prevention includes smoking cessation, blood pressure control, Mediterranean diet, regular exercise, appropriate medications, weight management, and knowing warning signs.

Broad preparation plans

To increase resilience during technological transition periods:

Be Informed: Develop situational awareness,

...
Softmax, Emmett Shear's new AI startup focused on "Organic Alignment"
by Chris Lakin — 2025-03-28T21:23:46.220Z
Link: https://www.corememory.com/p/exclusive-emmett-shear-is-back-with-softmax

A new AI alignment player has entered the arena.

Emmett Shear, Adam Goldstein and David Bloomin have set up shop in San Francisco with a 10-person start-up called Softmax. The company is part research lab and part aspiring money maker and aimed at figuring out how to fuse the goals of humans and AIs in a novel way through what the founders describe as “organic alignment.” It’s a heady, philosophical approach to alignment that seeks to take cues from nature and the fundamental traits of intelligent creatures and systems, and we’ll do our best to capture it here.

“We think there are these general principles that govern the alignment of any group of intelligent learning agents or beings, whether it's an ant colony or humans on a team or cells in a body,” Shear said, during his first interview to discuss the new company. “And organic alignment is the kind of alignment where a bunch of peers come together and find their role in a greater whole together where they maintain their individual identity.

“Organic alignment centers on this shared whole idea, which is opposed to the kind of alignment that you see from most foundational model companies that is very much about steering and control and direction. We think of that as hierarchical alignment.”

[…]

Also, https://softmax.com/about mentions collaboration with Michael Levin, Ken Wilber, Chris Fields, Ken Stanley, Denis Noble, Andrew Briggs, Jeff Clune, Erik Hoel, Ryan Smith, Center for the Study of Apparent Selves, Dalton Sakthivadivel, and Perry Marshall.
The Pando Problem: Rethinking AI Individuality
by Jan_Kulveit — 2025-03-28T21:03:28.374Z

Epistemic status: This post aims at an ambitious target: improving intuitive understanding directly. The model for why this is worth trying is that I believe we are more bottlenecked by people having good intuitions guiding their research than, for example, by the ability of people to code and run evals.

Quite a few ideas in AI safety implicitly use assumptions about individuality that ultimately derive from human experience.

When we talk about AIs scheming, alignment faking or goal preservation, we imply there is something scheming or alignment faking or wanting to preserve its goals or escape the datacentre.
If the system in question were human, it would be quite clear what that individual system is. When you read about Reinhold Messner reaching the summit of Everest, you would be curious about the climb, but you would not ask if it was his body there, or his mind, or his motivations to climb like the “spirit of mountaineering” or some combination thereof. In humans, all the answers are so strongly correlated that it does not make sense to ask.

But as we'll see, these human-centric assumptions often don't really fit the reality of AI systems.

We will start looking at a different domain: biology. Then, we'll look at AI systems, drawing on concepts from the "Three-Layer Model of LLM Psychology". At the end, we'll see how confusions about AI individuality can have significant safety implications.

Individuality in Biology

Individuality, even in relatively simple biological systems, turns out to be complex.

Consider the quaking aspen (Populus tremuloides). Pando, a famous aspen clone in Utah, is a grove of around 47,000 trees, all genetically identical, all growing from a single massive underground root system. In a sense, each tree is an individual—try chopping one down and the others won't fall. But in another sense, Pando is one individual—by some estimates the largest single organism on Earth, weighing in at 6,600 tons.

Then there's grafting. You can take a branch from an apple tree, graft it onto another apple tree's trunk, and voila—one tree, two sets of genes. Most commercial apples are actually grown this way, with the rootstock from one variety supporting the fruit-bearing branches of another. Is the resulting franken-tree one individual or two?

With mushrooms, you see individual mushrooms, individual fruiting...
Selection Pressures on LM Personas
by Raymond Douglas — 2025-03-28T20:33:09.918Z

TLDR:

Though we tend to have LMs play the role of a helpful assistant, they can in fact generate the responses of any persona you ask for
It’s not just explicit — the persona you get is also steered by implicit cues, and confined by what it’s easy for the LM to generate
These outputs are increasingly being spread all over the internet, and therefore fed back into the training data (including in indirect ways, like blog posts about the vibe of conversations), while also shaping user expectations
This creates a feedback loop, and a selection pressure on LM personas — for example, towards things that capture human attention and get discussed a lot
That could get real dicey

In slightly more detail

LMs are, very roughly, text prediction systems, which we happen to mainly use for predicting how a helpful assistant would respond to our queries. But in fact they can try to predict any text. In particular, they can predict how many different personas would respond to arbitrary inputs.

For example, you can just

ask ChatGPT to be Obama

, and it will do a pretty good job. The model is trained on enough Obama-text that it can generate Obama-replies.

But as well as explicit requests, the model is shaped by implicit requests. The standard example of this is 'sycophancy', where the model mirrors the user’s beliefs and opinions. Taking a

three-layer view

, probably some of this is the character deciding to play along, but I expect a lot of it is that the underlying predictive model is serving you a character which genuinely has those beliefs and opinions — because that’s the kind of character it expects you to be talking to. Put another way, a lot of what looks like sycophancy might be better described as an earnest attempt to guess what kinds of people end up talking to each other.

More broadly, you can think of the model as containing some kind of high-dimensional space of possible characters: any prior context (human- or model-generated) is effectively narrowing down what portion of the character-space you’re in.

This character-space is not uniform. It’s shaped by the training data, and by what it’s easy for the model to generate. ChatGPT does a good Obama because there’s a lot of Obama text on the internet, and when it’s doing “AI assistant”, this is also partly an extrapolation from all the AI assistant-y text on the internet — hence the fact that some AI assistants ...
AXRP Episode 40 - Jason Gross on Compact Proofs and Interpretability
by DanielFilan — 2025-03-28T18:40:01.856Z

YouTube link

How do we figure out whether interpretability is doing its job? One way is to see if it helps us prove things about models that we care about knowing. In this episode, I speak with Jason Gross about his agenda to benchmark interpretability in this way, and his exploration of the intersection of proofs and modern machine learning.

Topics we discuss:

Why compact proofs

Compact Proofs of Model Performance via Mechanistic Interpretability

What compact proofs look like

Structureless noise, and why proofs

What we’ve learned about compact proofs in general

Generalizing ‘symmetry’

Grading mechanistic interpretability

What helps compact proofs

The limits of compact proofs

Guaranteed safe AI, and AI for guaranteed safety

Jason and Rajashree’s start-up

Following Jason’s work

Daniel Filan (00:00:09):
Hello everybody. In this episode I’ll be speaking with Jason Gross. Jason is a researcher interested in mechanistic interpretability and software verification. During his PhD, he was on the team responsible for verifying the code used in encryption for the HTTPS protocol and he’s also on the development team of the Coq Proof Assistant. Links to what we’re discussing are in the description and you can read a transcript at axrp.net. You can also support the podcast at patreon.com/axrpodcast.

(00:00:35):
Well Jason, welcome to the podcast.

Jason Gross (00:00:37):
Thank you. I am excited to be here.

Why compact proofs

Daniel Filan (00:00:40):
Cool. So I guess we’re going to talk about this line of papers that starts with this paper “Compact Proofs of Model Performance via Mechanistic Interpretability”. You’re the lead author and then there’s a bunch of other authors that I don’t want to read on air, but can you give us a sense of just what’s the idea here? What’s the theme?

Jason Gross (00:01:02):
Okay. You can think of doing mech interp as compressing explanations of large models. And the takeaway from this paper is that you can use proofs to measure how much compression you get. That’s the super short version of it.

Daniel Filan (00:01:20):
This gets to this question I have about reading this paper, which is: there are a few things I could imagine you going for, right? One way of thinking about it is: we want to have a benchmark for how good a mechanistic interpretation is and our benchmark is, we’ll turn it into a proof and see some combi...
Share AI Safety Ideas: Both Crazy and Not. №2
by ank — 2025-03-28T17:22:22.814Z

AI safety is one of the most critical issues of our time, and sometimes the most innovative ideas come from unorthodox or even "crazy" thinking. I’d love to hear bold, unconventional, half-baked or well-developed ideas for improving AI safety. You can also share ideas you heard from others.

Let’s throw out all the ideas—big and small—and see where we can take them together.

Feel free to share as many as you want! No idea is too wild, and this could be a great opportunity for collaborative development. We might just find the next breakthrough by exploring ideas we’ve been hesitant to share.

A quick request: Let’s keep this space constructive—downvote only if there’s clear trolling or spam, and be supportive of half-baked ideas. The goal is to unlock creativity, not judge premature thoughts.

Looking forward to hearing your thoughts and ideas!

P.S. The AIs are moving fast, the last similar discussion was a month ago and was well received, so let's try again and see how the ideas changed.
How many times faster can the AGI advance the science than humans do?
by StanislavKrym — 2025-03-28T15:16:52.320Z

I hope that the reasoning in my two posts shows that the AGI has a chance to end up relying on the entire human-built energy industry just to solve as many problems (and hopefully even less) as the millions of humans who work there. On the other hand, the entire set of physicists is within half of an OOM from a million. If the AGI armed with the whole world's energy industry is worth tens of millions of scientists, then does it mean that it will invent the things just a hundred times faster? Is it likely that non-neuromorphic AGI won't accelerate the human progress at all?