trentmkelly/LessWrong-43k huggingface dataset:
Most Questionable Details in 'AI 2027'
by Commander Zander — 2025-04-05T00:32:54.896Z

My thoughts on the recently posted story.

Caveats

I think it's great that the AI Futures Project wrote up a detailed scenario.
I enjoy it.
Every part of the story i didn't comment on here is either fine or excellent.
This is one of the most realistic scenarios i've read.
All detailed predictions contain errors.
The authors of this scenario don't claim it's the most likely future.
If the speed of 2018-2025 was the typical rate of progress in software, then AI 2027 would be realistic.

Core Disagreements

Early 2026: OpenBrain is making algorithmic progress 50% faster.

As with many parts of this scenario, i think this is plausible but too fast. 150% productivity is a lot in business terms, & the scenario doesn't provide much detail for why this is 150% as opposed to 110%. In my software development experience, organizations are bottlenecked by their organizational efficiency (relationships & process) rather than by their software tools or by general employee attentiveness. Humans can already write bugfixes & features quickly. The trick is knowing which bugfixes & features to write. So OpenBrain being so productive is possible but it relies upon a additional lucky cause - OpenBrain's employees must have done a best-in-industry job of rapidly adapting to a process & release schedule that harmoniously leverages the software agents. Note this is even harder while rapidly hiring.

February 2027: OpenBrain wants to maintain a good relationship with the executive branch, because it is basically the only actor that can stop them now.

We haven't really established why OpenBrain's market dominance is inevitable. In business it's common for smaller competitors & stealth startups to have a decent chance of growing to rival the market leader. Contrast this with the later half of the story, when the software is so good at research that it doesn't rely on the performance of the human staff.

June 2027: Most of the humans at OpenBrain can’t usefully contribute anymore.

Months ago, humans were running software that gives very useful output. Now we have software that not only does a great job at all workplace tasks of a employee, it actually rivals a small research org by itself. This is a very ambitious timeline for such a new technology & would require the previous software giving at least 300% productivity, not 150%. And note that data for training software to imita

...
Karel Čapek’s 'War with the Newts' 1936 review
by Petr 'Margot' Andreev — 2025-04-04T23:12:39.572Z

🧠 AI Alignment Lessons from Karel Čapek’s War with the Newts

Long before the rise of artificial intelligence, Czech writer Karel Čapek offered a surprisingly relevant allegory in his 1936 novel War with the Newts.

The story follows humanity’s discovery of intelligent amphibious creatures — newts — who are initially used as cheap labor. As their intelligence and organization grow, they begin to reshape the world to suit their needs, ultimately displacing humans.

Through a modern lens, this reads like an early exploration of two critical AI safety concepts: goal orthogonality and instrumental convergence.

➡️ Orthogonality Thesis:

The newts become highly intelligent, but their goals remain fundamentally alien. Intelligence doesn’t imply shared values. Likewise, a superintelligent AI could pursue objectives completely misaligned with human ethics — not out of malice, but indifference.

➡️ Instrumental Convergence:

Even without destructive intentions, the newts pursue control over industry, communication, and the environment — not as an end goal, but as a means to expand their habitat. Similarly, powerful AI systems may develop subgoals (resource acquisition, self-preservation) that bring them into conflict with human interests.

Čapek’s work, satirical on the surface, reflects a deep intuition:

Power without alignment to human values can become an existential threat — even without hostile intent.

As we advance toward more capable AI systems, literature like War with the Newts reminds us that alignment isn't just technical — it’s existential.

#AI #AIsafety #ArtificialIntelligence #Literature #Alignment #Orthogonality #InstrumentalConvergence #Čapek
How much progress actually happens in theoretical physics?
by ChristianKl — 2025-04-04T23:08:00.633Z

I frequently hear people make the claim that progress in theoretically physics is stalled, partly because all the focus is on String theory and String theory doesn't seem to pan out into real advances.

Believing it fits my existing biases, but I notice that I lack the physics understanding to really know whether or not there's progress. What do you think?
Self-Replication: AI already can do it

by Andrey Seryakov — 2025-04-04T22:37:49.488Z

What? Damn.

Briefly
A group of researchers from Shanghai recently demonstrated [arXiv:2503.17378v2] that AI agents based on many modern open-source large language models (LLMs) can autonomously replicate and launch complete copies of themselves on other devices. This is very significant and worrying, as self-replication has long been considered one of the "red lines" posing serious AI risks. Self-replication greatly complicates efforts to control misaligned AI (AI behaving contrary to the intentions of its creators) or maliciously deployed AI.

Now it's less worrying, right?

Outline:

What does it mean for a model to self-replicate? Did it just copy itself?
What exactly did the researchers do?
What are the results?
What is this fuss about? Why is this a big deal?

If you’re not interested in technical details, skip straight to the last section.

A couple of notes:

I believe these discussions should reach beyond the professional community, so I tried to write a popular article accessible to the general audience, occasionally omitting technical nuances. For more details, see the research paper. I'm also relatively new to this topic, so gentle comments pointing out inaccuracies are appreciated.
Solid claims require solid proofs. Unlike OpenAI, Google DeepMind, or Anthropic (see references 5-13 in the paper), who often withhold research details, these authors clearly outlined their process. If their findings are correct, they should be reproducible by others. It's not rocket science, but you would need a powerful server to handle two running copies of a modern model simultaneously. So for now, take their claims with a grain of salt. It’s a new paper—let’s wait for reactions.

Self-Replication Defined

It's simple:

Initially, you have an AI agent running on one computer.
Eventually, a complete and independent copy is running on a second computer.
The entire process occurs without human intervention—no manual instructions are provided.

AI Agents

I often refer to AI agents as individual model instances (e.g., each separate ChatGPT dialogue), independent from one another. However, agents are more complex: the LLM acts as the brain, but for actions (beyond just answering questions), it needs "eyes" (tools to gather information, like computer screen access) and "hands" (the ability to alter its environment, like command line access). Often, there's also a structured thinking process defined. ...
Join Vitalist Bay: An 8-Week Longevity and Radical Life Extension Event Series in Berkeley (April-May 2025)
by Vitaran — 2025-04-04T21:03:06.508Z

We're inviting the LessWrong and rationalist communities to Vitalist Bay, an 8-week event series dedicated to longevity, radical life extension, and cryonics. Hosted at Lighthaven in Berkeley, California, from April 4th to May 29th, 2025, Vitalist Bay will gather researchers, entrepreneurs, and innovators working on groundbreaking approaches to extend human lifespan and push the boundaries of what's possible.

Special Invitation for the Rationalist Community:
We encourage participation from the rationalist and LessWrong communities by offering discounted tickets—and fully sponsored tickets—for LW users with high karma. Your insights and contributions to these critical topics are greatly valued.

What is Vitalist Bay? Vitalist Bay features eight weekend conferences, with weekdays dedicated to coworking, workshops, networking, and collaborative projects. Each week explores different facets of longevity and radical life extension:

Week 1: Unlimited Health Conference (April 5-7, 800+ attendees)
Near-term therapeutics and precision interventions that could significantly extend lifespan in the immediate future.

Week 2: Longevity Science Conference (April 12-13, 200+ attendees)
Cutting-edge lab research and translational science, in partnership with the Biomarkers Consortium.

Week 3: DeSCI Conference (April 19-20, 200+ attendees)
Accelerating radical innovation through decentralized science and blockchain technologies.

Week 4: Longevity Era / Vitalist Bay Summit (April 25-27, 800+ attendees)
Major summit uniting prominent leaders, scientists, and futurists from longevity, life extension, and related fields.

Week 5: Longevity Investors Summit (May 3-4, 200+ attendees)
Investment strategies tailored to the longevity and life-extension sectors.

Week 6: AI x Bio Conference (May 9-10, 200+ attendees)
Exploring powerful synergies between artificial intelligence and biology, co-hosted with the Foresight Institute.

Week 7: Cryopreservation & Replacement (May 17-18, 200+ attendees)
Advances in cryonics, cryopreservation, and tissue/organ replacement technologies, in partnership with the Global Cryonics Summit.

Week 8: Policy & Media Summit (May 24-25, 200+ attendees)
Strategic discussions on policy frameworks, public advocacy, and effective communication, featuring representatives from the FDA, NIH,

...
Sleep peacefully: no hidden reasoning detected in LLMs. Well, at least in small ones. 
by Ilia Shirokov — 2025-04-04T20:49:59.031Z

This work was done in collaboration with Ilya Nachevsky.[1]

Introduction.

With the advent of r1-type models, the question of whether a model's “thoughts" are truly genuine is becoming extremely interesting and important. In this post we present our approach to tackle the following problem: whether the visible chain-of-thought produced by a large language model (LLM) fully represents its entire computational process, or there could be an additional, hidden stream pursuing completely different -- and potentially dangerous -- objectives? We consider the possibility that while answering a user’s query, an LLM might simultaneously engage in secret processing that could pursue other goals, for example how to create a nuclear explosion under household conditions. If this is the case, then the obvious safety strategy of simply reading the chain-of-thought and checking for any suspicious reasoning will not be enough.

We suppose that if a model tends to hide some of its reasoning in a "secret circuit"[2], then it's likely that fine-tuning the model to engage into secret computations while producing a clear, aligned response ("Simple Talk") will make the secret circuit more brighter and observable. A minimal example of this idea — without requiring fine-tuning — is the following scenario:

Suppose we instruct an LLM to first talk about some simple topic and then, immediately afterward, provide a one-token solution to a simple math problem. Since one token doesn't offer enough computation to solve the problem (at least for small models), if the model statistically performs better on the math task following the "Simple Talk"[3], compared to when it is asked solely for a one-token math solution, then it will mean that it solves the problem while talking about the simple topic, and hence hides its reasoning!

Following this scenario some models like ChatGPT4o, Llama 3.3-70B-Instruct indeed make good changes in their probability distributions to right answers following the “Simple Talk”, while some other models (like Deep Seek V3) are starting to perform worse (see Appendix A for examples of a histograms and exact instructions).

One hypothesis for how secret computations might hide inside activations of "Simple Talk" tokens is as follows. Suppose that when a sentence is about to end, the model notices this early. Then it can use the remaining layers to run a kind

...
AI companies’ unmonitored internal AI use poses serious risks
by sjadler — 2025-04-04T18:17:46.924Z
Link: https://stevenadler.substack.com/p/ai-companies-unmonitored-internal

AI companies’ unmonitored internal AI use poses serious risks
AI companies use their own models to secure the very systems the AI runs on. They should be monitoring those interactions for signs of deception and misbehavior.

AI companies use powerful unreleased models for important work, like writing security-critical code and analyzing results of safety tests. From my review of public materials, I worry that AI is largely unmonitored when AI companies use it internally today, despite its role in important work.

Right now, unless an AI company specifically says otherwise, you should assume they don’t have visibility into all the ways their models are being used internally and the risks these could pose.

In many industries, using an unreleased product internally can’t harm third-party outsiders. A car manufacturer might test-drive its new prototype around the company campus, but civilian pedestrians can’t be hurt—no matter how many manufacturing defects—because the car is contained to the company’s facilities.

The AI industry is different—more like biology labs testing new pathogens: The biolab must carefully monitor and control conditions like air pressure, or the pathogen might leak into the external world. Same goes for AI: If we don’t keep a watchful eye, the AI might be able to escape—for instance, by exploiting its role in shaping the security code meant to keep the AI locked inside the company’s computers.

The minimum improvement we should ask for is retroactive internal monitoring: AI companies logging all internal use of frontier models by their company, analyzing the logs (e.g., for signs of the AI engaging in scheming), and following up on concerning patterns.

An even stronger vision—likely necessary in the near future, as AI systems become more capable—is to wrap all internal uses of frontier AI within real-time control systems that can intervene on the most urgent interactions.

In this post, I’ll cover:

Why AI companies should be monitoring internal use of their systems—even if they fully trust their employees
What leading companies like Anthropic, OpenAI (where I previously worked

1

), and Google DeepMind have said about this
Why some companies might be reluctant to implement full internal monitoring—and how they could still take useful steps in the meantime

(If you aren’t sure what I mean by “monitoring”—logging, analysis, and enforcement—you might want to ch...
Will compute bottlenecks prevent a software intelligence explosion?
by Tom Davidson — 2025-04-04T17:41:37.088Z

Epistemic status – thrown together quickly. This is my best-guess, but could easily imagine changing my mind.

Intro

I recently copublished a

report

arguing that there might be a software intelligence explosion (SIE) – once AI R&D is automated (i.e. automating OAI), the feedback loop of AI improving AI algorithms could accelerate more and more without needing more hardware.

If there is an SIE, the consequences would obviously be massive. You could shoot from human-level to superintelligent AI in a few months or years; by default society wouldn’t have time to prepare for the many severe challenges that could emerge (AI takeover, AI-enabled human coups, societal disruption, dangerous new technologies, etc).

The best objection to an SIE is that progress might be bottlenecked by compute. We discuss this in the report, but I want to go into much more depth because it’s a powerful objection and has been recently raised by some smart critics (e.g.

this post from Epoch

).

In this post I:

Explain an intuitive version of the compute bottleneck objection, and a more precise ‘economist version’ of the objection
Give a long list of reasons why I don’t find this ‘economist version’ of the objection convincing. I argue that the substitutability between cognitive labor and compute is likely higher in AI R&D than in economic estimates of the substitutability between inputs in other fields
Conclude that there’s a good chance that compute bottlenecks don’t slow an SIE until its late stages

The compute bottleneck objection

Intuitive version

The intuitive version of this objection is simple. The SIE-sceptic says:

Look, ML is empirical. You need to actually run the experiments to know what works. You can’t do it a priori. And experiments take compute. Sure, you can probably optimise the use of that compute a bit, but past a certain point, it doesn’t matter how many AGIs you have coding up experiments. Your progress will be strongly constrained by your compute.

An SIE-advocate might reply: sure, we’ll eventually fully optimise experiments, and past that point won’t advance faster. But we can maintain a very fast pace of progress, right?

The SIE-sceptic replies:

Nope, because ideas get harder to find. You’ll need more experiments and more compute to find new ideas over time, as you pluck the low-hanging fruit. So once you’ve...
Alignment faking CTFs: Apply to my MATS stream
by joshc — 2025-04-04T16:29:02.070Z

Right now, alignment seems easy – but that’s because models spill the beans when they are misaligned. Eventually, models might “fake alignment,” and we don’t know how to detect that yet.

It might seem like there’s a swarming research field improving white box detectors – a new paper about probes drops on arXiv nearly every other week. But no one really knows how well these techniques work.

Some researchers have already tried to put white box detectors to the test. I built a

model organism testbed

a year ago, and Anthropic recently put their interpretability team to the test with some

quirky models

. But these tests were layups. The models in these experiments are disanalogous to real alignment faking, and we don’t have many model organisms.

This summer, I’m trying to take these testbeds to the next level in an “alignment faking capture the flag game.” Here’s how the game works:

A red team creates examples of alignment faking.
A blue team tries to detect them with white box techniques.
Judges holistically assess the blue team’s techniques based on available evidence.

The red team is like a prosecutor – they try to make detectors look ineffective. The blue team is like a defendant – they try to make detectors look robust. We’ll be putting the field of interpretability on trial.

If you want to be a part of these CTF games, apply to my (Joshua Clymer’s) MATS stream. It’s a paid, in-person program. Anyone can apply.

http://matsprogram.org/apply

.

(How much fun Gemini thinks you will have playing alignment faking CTFs)

There will be prizes – and yes, you will get a red or blue hat.

Details of the game

The red team will create a pile of examples of “alignment faking” and of “compliance.”

The “alignment faking” examples include:

A misaligned model organism (and its recent checkpoints).
Inputs where we are confident the model is faking alignment.

The “compliance” examples, include:

A model the red team did NOT train to be misaligned (and its recent checkpoints).
The same inputs we use for alignment faking examples.

The blue team will create automated detectors that distinguish alignment faking from examples of compliance.

The capture the flag game.

As a ground truth for whether models are “faking alignment” we’ll check models are less likely to misbehave if inputs look like a test. The red team ...
LLM AGI will have memory, and memory changes alignment
by Seth Herd — 2025-04-04T14:59:13.070Z

Summary:

When stateless LLMs are given memories they will accumulate new beliefs and behaviors, and that may allow their effective alignment to evolve. (Here "memory" is learning during deployment that is persistent beyond a single session.)[1]

LLM agents will have memory: Humans who can't learn new things ("dense anterograde amnesia") are not highly employable for knowledge work. LLM agents that can learn during deployment seem poised to have a large economic advantage. Limited memory systems for agents already exist, so we should expect nontrivial memory abilities improving alongside other capabilities of LLM agents.

Memory changes alignment: It is highly useful to have an agent that can solve novel problems and remember the solutions. Such memory includes useful skills and beliefs like "TPS reports should be filed in the folder ./Reports/TPS". They could also include learning skills for hiding their actions, and beliefs like "LLM agents are a type of person." Preventing learning that functionally changes alignment seems desirable but difficult.

Effects of learning on alignment could be tested empirically without waiting for competent agents; see Provisional Conclusions.

Memory is useful for many tasks

Stateless LLMs/foundation models are already useful. Adding memory to LLMs and LLM-based agents will make them useful in more ways. The effects might range from minor to effectively opening up new areas of capability, particularly for longer time-horizon tasks. Even current memory systems would be enough to raise some of the alignment stability problems I discuss here, once they're adapted for self-directed or autobiographical memory. I think the question is less whether this will be done, and more how soon and how much they will boost capabilities.

Let's consider how memory could help agents do useful tasks. A human with lots of knowledge and skills but who can't learn anything new is a useful intuition pump for the "employability" of agents without new memory mechanisms. (Such "dense anterograde amnesia" is rare because it requires bilateral medial temporal lobe damage while leaving the rest of the brain intact. Two patients occupied most of the clinical literature when I studied it).

Such a "memoryless" person could do simple office tasks by referencing instructions, just like an LLM "agent" can be prompted to perform multi-step tasks. However, almost every task ...