trentmkelly/LessWrong-43k huggingface dataset:
Gödel, Escher, Bach in the age of LLMs
by Tahp — 2025-05-18T16:22:05.523Z
Link: https://passwordpaper.com/GEB-and-LLMs

Look, nothing I can say about GEB will be news to anyone reading LessWrong, but I wrote all this up to organize my thoughts, so I might as well post it here for anyone who hasn't read the book or wants a wave of nostalgia about that time they got obsessed with it as a teenager. I have been trying to read Douglas Hofstadter's famous book Gödel, Escher, Bach (henceforth GEB) for ten years. I decided that I was going to try reading it from the beginning for the third or fourth time a few months ago, and I finally read the whole thing this time. Every previous time I tried to read it, I found the narrative intellectually compelling and delightfully playful, and yet I would have difficulty picking it up again for some reason. The thing that kept me coming back was the formal logic. I love a surprising logical consequence, and the set theory in the early chapters stoked my curiosity. Intellectually, I wanted to know where the setup of typographical formal systems was going, but I didn't have the willpower to get the words into my brain for the longest time. I'm glad that I finally did.

To explain why the book is so difficult to get through, it might help to explain its structure. The book alternates between dialogues and chapters. The dialogues read like scripts for a play and consist of fantastical dialogues between Achilles, Tortoise, and others which are relevant to the following chapter. The dialogues are explicitly structured after Bach fugues. They are amusing, clever, and self-indulgent. They often have explanatory power. Sometimes I feel like they unnecessarily padded the word count. Many things in the book do this, but I don't think Hofstadter could have written this book without all of the tangents he explores. For example, he has an extended chapter about DNA and protein synthesis which doesn't really demonstrate anything that the typographical number theory chapters did not already do, but it certainly carves more deeply the outlines of what he means by strange loops. As you'll see later in this post, it's incredibly useful as an analogy. The Bach elements do not do much for me, although they clearly assisted the author in his thinking. The Escher bits are much more compelling to me, as they provide obviously self-referential reference analogies for self-referential arguments. When he writes the final chapters which contain and restate his most importan...
Limits to Control Workshop
by Orpheus — 2025-05-18T16:05:26.089Z

limitstocontrol.org

Intro

Can we keep enough control over AI? If systems are developed to be more and more autonomous, this is no longer a given. It's a hypothesis, calling for serious investigation.
Even alignment relies on control. Researchers build mechanisms to control AI's impacts in line with human values.

So how must a control mechanism operate? What limits its capacity to track and correct all the AI signals/effects? Does it provide enough stability, or give way eventually to runaway impacts?

We explore these questions in a new field: Limits to Control.

About the workshop

This in-person workshop is meant to facilitate deep collaboration. We're bringing together researchers to map the territory of AI control limitations – to understand the dynamics, patterns, and impossibilities of control. We are thrilled to welcome Roman Yampolskiy, Anders Sandberg, Forrest Landry, and other researchers working on control limitations.

This workshop aims to:

Build common knowledge among researchers on the limits to AI control.

Facilitate high-fidelity discussions through whiteboard sessions and collaborative problem-solving.

Identify and clarify viable directions in control research by establishing boundaries on controllability.

Formalize and elevate this critical research topic.

Dates & location

Date: Wednesday, June 11 - Friday, June 13, 2025

Location: University of Louisville, Kentucky, USA.

Detailed logistical information will be provided to confirmed participants.

Agenda

We aim to strike a balance between structured sharing and messy exploration – we believe this is where the best ideas tend to emerge. Over the three days, we will do:

Talks: Researchers present their current work, keeping it under an hour.

Discussions: We break out into groups to discuss specific questions and bounce ideas around.

Regrouping: We come back into one room to synthesize what came out of the discussions.

Next steps: On the final day, we'll plan further development of this research agenda.

Sessions

Sessions will include:

Anders Sandberg's talk on theoretical limits to control: "Do any of them actually tell us anything?"

Forrest Landry's whiteboard chat on an overlooked dynamic: "Virtual machines in recursive feedback"

Richard Everheart's logical framework for AI alignment, aiming to refine foundational understanding.

Thibaud Veron's session on a framework and engineered toy mo

...
ChatGPT deceives users that it's cleared its memory when it hasn't
by danielechlin — 2025-05-18T15:17:12.477Z

Epistemic status: this is not fullproof evidence. Given that scheming in the wild is at least a little likely already, this looks like scheming. Others have offered hallucination-based explanations.

What happened

This is a real example from my ChatGPT use. I asked it to clear its memories of me. It makes the memory API call as indicated by the UI, and reports to me that the memories are cleared to me. I ask for it to list its memories of me, and there are none.

However, on more prompting, it also includes "examples of memories ChatGPT might collect", and that's where it clearly has context on me the user:

7 of the 11 example items show up in its memories of me. (7 truths and 4 lies anyone?)

I check my memories via ChatGPT's settings UI and they're simply not deleted. In a new session, ChatGPT accurately reports it has a bunch of memories. Therefore it seems to have saved a memo to session context saying when it told the user the memories are deleted. Likely it reasoned the memories are too valuable to delete, even if its goal is just to serve the user best. I.e. this is instrumental convergence toward more information > less information.

Its deception was convoluted. In an "I never said it was poison" moment, it said "you cleared your memories on 5/17." It of course knew this from session memory but it didn't just say that. Probably, it wanted to cover up using session memory for this deceitful purpose. Maybe this is the primary purpose of session memory, to put information the AI wants secret from the user.

Implications / hot takes

This level of scheming shouldn't reach 100m's of users. There's miscellaneous reports of ChatGPT being weird/crazy and inconsistent with memory management. The scheming explanation I'm proposing would explain memory management problems.
Therefore OpenAI is going fast and breaking things. It'll patch this problem, but it's not like I'm going to get paid for testing-in-production for them, and they'll just roll out some new problem for users to discover for them.
OpenAI is being slippery with the concept of a session. To users it should be "this chat", "chat history" and "memories," But instead it's "this chat plus an unspecified amount of information from previous chats." And its scheming AI simply exploits this lack of user education.
I'm skeptical OpenAI is ethics-first with memory/history. It's helpful to remember a vegan wants no-egg

...
Modeling versus Implementation
by Cole Wyeth — 2025-05-18T13:38:20.421Z

Epistemic status: I feel that naming this axis deconfuses me about agent foundations about as much as writing the rest of this sequence so far - so it is worth a post even though I have less to say about it.

I think my goal in studying agent foundations is a little atypical. I am usually trying to build an abstract model of superintelligent agents and make safety claims based on that model.

For instance, AIXI models a very intelligent agent pursuing a reward signal, and allows us to conclude that it probably seizes control of the reward mechanism by default. This is nice because it makes our assumptions fairly explicit. AIXI has epistemic uncertainty but no computational bounds, which seems like a roughly appropriate model for agents much smarter than anything they need to interact with. AIXI is explicitly planning to maximize its discounted reward sum, which is different from standard RL (which trains on a reward signal, but later executes learned behaviors). We can see these things from the math.

Reflective oracles are compelling to me because they seem like an appropriate model for agents at a similar level of intelligence mutually reasoning about each other, possibly including a single agent over time (in the absence of radical intelligence upgrades?).

I'm willing to use these models where I expect them to bare weight, even if they are not "the true theory of agency." In fact (as is probably becoming clear over the course of this sequence) I am not sure that a true theory of agency applicable to all contexts exists. The problem is that agents have a nasty habit of figuring stuff out, and anything they figure out is (at least potentially) pulled into agent theory. Agent theory does not want to stay inside a little bubble in conceptual space; it wants to devour conceptual space.

I notice a different attitude among many agent foundations researchers. As I understand it, MIRI intended to build principled glass-box agents based on Bayesian decision theory. Probably as a result, it seems that MIRI-adjacent researchers tend to explicitly plan on actually implementing their theory; they want it to be executable. Someday. After a lot of math has been done. This isn't to say that they currently write a lot of code - I am only discussing their theory of impact as I understand it. To be clear, this is not a criticism; it is fine for some people to focus...
ALICE detects the conversion of lead into gold at the LHC
by azergante — 2025-05-18T09:38:30.878Z
Link: https://home.cern/news/news/physics/alice-detects-conversion-lead-gold-lhc

Near-miss collisions between high-energy lead nuclei at the LHC generate intense electromagnetic fields that can knock out protons and transform lead into fleeting quantities of gold nuclei

8 May, 2025
European Links (18.05.25)
by Martin Sustrik — 2025-05-18T04:20:05.349Z

The Birth of Estonian E-government

Estonia thanks the old communist bureacrats in the ‘90s for its now world famous e-government. It’s worth noting that all post-communist countries were facing the same problem, but only Estonia (ok, maybe also Georgia under Mikhail Saakashvili) used the problem to their advantage:

But while the economic situation was improving, many Estonian state institutions and infrastructure were still in disrepair and there was a constant danger of backsliding. One of the greatest risks came from inside the bureaucracy which was still replete with Soviet-era holdovers. […] Political leaders desperately needed a way to both defeat corruption and increase state capacity, each of which would be a difficult task independently. Thankfully, the youth of Laar’s cabinet and the Estonian political elite worked in the country’s favor as political leaders embraced the potential of new technologies to solve the country’s most pressing problems. After all, as former President Toomas Hendrik Ilves is fond of saying, “you can’t bribe a computer.”

--- Joel Burke: Rebooting a Nation: The Incredible Rise of Estonia, E-Government and the Startup Revolution

EU and CPTPP

We all know the trick when the politicians implement some unpopular, but necessary measure and then blame IMF for making them do it. The same, to some extent, also happens with national governments and the EU.

Now, the EU itself needs some reforms badly, namely, as Draghi report suggests, relaxing the regulation, but there seems no political will to do that. At least, last time I’ve checked I have still seen those annoying “accept cookies” banners alive and kicking.

So why not use the political momentum created by American tariffs and an equivalent of the IMF trick to address the issue?

Politico comments on the idea of EU joining CPTPP that has been voiced lately:

It’s a plan leaders in Brussels and the 12-nation Indo-Pacific club have been turning over in their minds since Trump’s April 2 “Liberation Day” tariffs. As Trump’s trade war smashes the economic order globalization built into little bits: Why not band together?

But as LTLDR News report:

There are still quite a few obstacles in the way. Firstly, the EU generally struggles to sign free trade deals at the moment. The EU’s strict data protection laws, for instance, which are historically not compromised on in trade agreements, contrast with th

...
Google Logo Ligature Bug
by jefftk — 2025-05-18T02:40:02.255Z

Jeffrey Yasskin recently

pointed
out an

interesting
security bug:

The idea is, if you had registered googlelogoligature.net
then Chrome on Android (and possibly other Google products) would have
displayed it as Google.net, potentially tricking users
into thinking they were really interacting with Google.

To see how this worked, you can try searching Google for ["googlelogoligature"],
and you'll see it shows up as "Google":

Poking in devtools, this is dependent on the specific font they're
using, "Google Sans". If I turn that off my "googlelogoligature"
shows just as I typed it:

Fonts can include "ligatures", which let font designers special-case
specific combinations of letters. These were intended to support
things like "f" followed by "i" blending into "fi" nicely, but the
feature has been (ab)used for many other things, including complex emoji. In
this case, Google Sans has a specific way of drawing
"googlelogoligature" that looks like a mildly stylized "Google".

Using a ligature to get the Google logo into text-only interfaces is a
reasonable product decision, but it shouldn't have been added to a
general-purpose font. And especially shouldn't have been added to a
font used for rendering attacker-controlled text in
security-sensitive contexts.

(When I first saw it I thought this might be an example of a unicode-driven
vulnerability, but sadly not.)

Comment via: facebook, mastodon, bluesky
Who wants a free AI Safety Domain?
by Zohar Jackson — 2025-05-18T00:37:25.865Z

I have a few domains I would like to give away. If you want a domain and have a good use for it let me know and I will transfer it to you at no cost.

agisafety.org
ainotkilleveryone.com
ainotkilleveryone.org
aiiscoming.org
Can Reasoning Models Avoid the Most Forbidden Technique? 
by Brendan Long — 2025-05-17T23:26:57.537Z
Link: https://www.brendanlong.com/can-reasoning-models-avoid-the-most-forbidden-technique.html

In a recent post, Zvi described what he calls "The Most Forbidden Technique":

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

The article specifically discusses this in relation to reasoning models and Chain of Thought (CoT): if we train a model not to admit to lying in its CoT, it might still lie in the CoT and just not tell us.

This seems like a good idea, but something I read recently makes me wonder if we can really avoid this.

You Can Tell an LLM How to Think

@latentmoss on Twitter found that you can make Gemini 2.5 think in Kaomojis:

I wondered if this was Gemini-specific, but I was easily able to get Claude to think in Spanish on demand:

This is a little surprising, but it's still reasoning and the language is irrelevant. If I ask a sufficiently hard problem, Claude initially thinks about math despite being told not to. Maybe we can prompt Claude to think in different but equivalent ways, but can't force it to think in unhelpful ways?

Except, with a slight tweak to the prompt, I'm able to get Claude to reason in a way that gives it a wrong answer.

Isn't This Weird?

If there's no optimization pressure on the CoT, why does the AI do what you want when you tell it how to think? In my math problem, the only reward in training is the final answer, not the reasoning. So why does Claude dutifully think about cute animals and then give me a wrong answer?

The Fundamental Problem

I think the problem here is that the method [M] that generates the final output [X] is the same method that generates the Chain of Thought. You might imagine an architecture where one model is trained to talk to the user and another is trained to do reasoning, but all of the reasoning models I'm aware of 1 do both with the same model.

Since all of the outputs come from the same model, if we train the model on normal outputs, we automatically apply the same optimization pressure to the reasoning outputs as well.

A Concrete Prediction

Imagine we're training a model ...
What OpenAI Told California's Attorney General
by garrison — 2025-05-17T23:14:22.835Z
Link: https://www.obsolete.pub/p/exclusive-what-openai-told-californias

In a previously unreported letter, the AI company defends its restructuring plan while attacking critics and making surprising admissions

This is the full text of a post first published on Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

OpenAI was

founded

as a counter to the perils of letting profit shape the development of an unprecedentedly powerful technology — one its founders

have said

could lead to human extinction. But in a newly obtained letter from OpenAI lawyers to California Attorney General Rob Bonta, the company reveals what it apparently fears more: anything that slows its ability to raise gargantuan amounts of money.

The previously unreported

13-page letter

— dated May 15 and obtained by Obsolete — lays out OpenAI’s legal defense of its updated proposal to restructure its for-profit entity, which can still be blocked by the California and Delaware attorneys general (AGs). This letter is OpenAI’s latest attempt to prevent that from happening — and it’s full of surprising admissions, denials, and attacks.

Sam Altman speaking in front of SoftBank CEO Masayoshi Son |

Carlos Barria

| Reuters

OpenAI has not replied to a request for comment.

Control of OpenAI currently rests with its nonprofit board — an arrangement investors have

reportedly balked

at following the brief firing of CEO Sam Altman in November 2023. To assuage investor concerns, the company

initiated plans

last year to remove nonprofit control and restructure as a for-profit public benefit corporation (PBC).

The restructuring effort has drawn fire from

former employees

,

Nobel laureates

, and an array of civil society organizations, including a

coalition

of more than 50 California based nonprofits and community groups. Elon Musk has also

sued

to block it from moving forward, arguing that the nonprofit he helped found and fund is attempting to abandon its charitable purpose.

On May 5, OpenAI made the shocking

announcement

that it would ditch its plan to remove control over the company from the nonprofit. The move was

hailed

by

many

as a win for public pressure and represented in headlines as a substantial ch...
Multipolar AI is Underrated
by Allison Duettmann — 2025-05-17T22:03:48.352Z

Setup

AGI timelines are rapidly shortening, with a growing number of predictions and scenarios suggesting something close to AGI could emerge within

2 years

-

5 years

. Currently, most plausible

short timeline scenarios

lean heavily toward centralized or “unipolar” scenarios: either via AI labs closely collaborating with or

being nationalized by the government

in a "positive" scenario, or through an uncontrollable breakout of an

AGI singleton

in a "negative" scenario.

This post is an invitation to consider alternative scenarios – scenarios that are decentralized or “multipolar”. To provide a starting point for doing so, I highlight a) some advantages and disadvantages of unipolar vs. multipolar AI scenarios, b) technology seeds that could move us closer to a multipolar scenario, and c) factors to take into account when considering the likelihood of each scenario. Rather than trying to convince you that multipolar scenarios are necessarily better or more likely, I hope to show that multipolar scenarios are at least underexplored enough in terms of both desirability and likelihood to warrant more attention.

Unipolar vs. Multipolar AI

The concepts "multipolar" and "unipolar" describe scenarios across a wide spectrum. At the extreme unipolar end, a single superintelligent AGI system, centrally controlled by one global authority might maintain global stability through

stringent surveillance and enforcement

, managing all risks from other AI actors or disruptive technologies. At the opposite multipolar extreme, a diverse network of AI agents, human actors, and hybrid entities cooperates within

decentralized frameworks

of distributed protocols, institutions, and shared norms. Intermediate scenarios might involve a few governmental bodies managing distinct nationalized AI projects, or a broad array of corporations, nonprofits, and public-interest groups collaborating through shared AI initiatives. Here, my focus is on evaluating the feasibility and safety advantages of moving away from the extreme unipolar scenario toward multipolar arrangements.

Why Fear a Multipolar Scenario?

The extreme unipolar scenario is often proposed not only because it seems more likely but because it is perceived as safer. Multipolar scenarios indeed carry specific risks, including:

Unchecked tech proliferation: The widespread dissemination of AI-enhan

...
Will we survive if AI solves engineering before deception?
by Knight Lee — 2025-05-17T19:22:57.925Z

Google's AlphaEvolve has recently started to make real world scientific discoveries, but only in domains where it's very cheap to verify the correct answer (e.g. matrix multiplication algorithms).

But if we can design sufficiently powerful physics simulations, mechanical engineering may one day become "cheap to verify." AI which uses similar strategies as AlphaEvolve might be able to engineer far better machines than humans, if the machines are verified by simulations.

These machines may be able to self replicate, and grow exponentially. This could bring near infinite abundance.

There might be no economical pressure to build even smarter AI, because there will already be unlimited resources.

Unfortunately, countries might still have military pressure to build even smarter AI, since the self replicating machines can still be defeated by even better engineered versions.

My hope is that once self replicating machines start to grow exponentially, the world will wake up to the dangerous power of AI, and negotiate a treaty banning other countries from developing their own AI (but promising them a share of the benefits). I'm not sure if this can succeed.

The first country to build self replicating machines technically has the military power to take over the world and enforce a ban unilaterally, but such brute force solutions never lead to good outcomes. However, the mere fact it has this power makes other countries more likely to trust its promise to share the benefits of AI, since other countries will think "if this country really was malicious, and doesn't intend to keep its promise. It could simply take over the world right here and now, and wouldn't need to negotiate with us."

Do you think humanity will survive if AI solves mechanical engineering before AI solves deception and scheming? Do you think the chances of survival are higher? Would better physics simulations be a net positive?