trentmkelly/LessWrong-43k huggingface dataset:
What's Wrong With the Simulation Argument?
by AynonymousPrsn123 — 2025-01-18T02:32:44.655Z

In LessWrong contributor Scott Alexander's essay, Espistemic Learned Helplessness, he wrote,

Even the smartest people I know have a commendable tendency not to take certain ideas seriously. Bostrom’s simulation argument, the anthropic doomsday argument, Pascal’s Mugging – I’ve never heard anyone give a coherent argument against any of these, but I’ve also never met anyone who fully accepts them and lives life according to their implications.

I can't help but agree with Scott Alexander about the simulation argument. No one has refuted it, ever, in my books. However, this argument carries a dramatic, and in my eyes, frightening implication for our existential situation.

Joe Carlsmith's essay, Simulation Arguments, clarified some nuances, but ultimately the argument's conclusion remains the same.

When I looked on Reddit for the answer, the attempted counterarguments were weak and disappointing.

It's just that, the claims below feel so obvious to me:

It is physically possible to simulate a conscious mind.
The universe is very big, and there are many, many other aliens.
Some aliens will run various simulations.
The number of simulations that are "subjectively indistinguishable" from our own experience far outnumbers authentic evolved humans. (By "subjectively indistinguishable," I mean the simulates can't tell they're in a simulation. )

When someone challenges any of those claims, I'm immediately skeptical. I hope you can appreciate why those claims feel evident.

Thank you for reading all this. Now, I'll ask for your help.

Can anyone here provide a strong counter to Bostrom's simulation argument? If possible, I'd like to hear specifically from those who've engaged deeply and thoughtfully with this argument already.

Thank you again.
Your AI Safety focus is downstream of your AGI timeline
by Michael Flood — 2025-01-17T21:24:11.913Z

Cross-posted from Substack

Feeling intellectually understimulated, I've begun working my way through Max Lamparth's CS120 - Introduction to AI Safety. I'm going to use this Substack as a kind of open journaling practice to record my observations on the ideas presented, both in the lectures and in the readings.

The reading for the first week is Dobbe, Gilbert, and Mintz's 2021 paper "Hard Choices in AI Safety." The paper makes the argument that AI Safety and Governance need to be situated in a 'sociotechnical' milieu - that there is a lot to be learned from sociological studies of complex technology and systems, and that this should flesh out and expand the narrow (from the authors' POV) focus on technical solutions that they attribute to much contemporary AI Safety research. Essentially, they advocate for thinking about AI Safety and Governance as political, social issues as opposed to narrowly technical, mathematical ones.

The authors do not at any point engage with ideas about Artificial General Intelligence or the idea of creating generally capable AI agents and how transformative this could be. Rather, their examples are couched in terms of autonomous vehicles, content curation systems, and algorithms perpetuating bias and inequalities - call it Everyday AI. Essentially, they treat AI as equivalent to other transformative technologies like electricity or the Internet, rather than something unprecedented. They imagine AI deployment as the selection and adaptation of a system for a given purpose - a police department adopts a predictive algorithm for patrol routes, a school system purchases a personalized learning curriculum - and then think about how to engage with and gather feedback from the communities affected while mitigating power imbalances between developers, users, and other stakeholders affected by the system's use. They're not thinking of ubiquitous, powerful, autonomous AI agents available anytime and anywhere there is an internet connection, pursuing autonomous goals.

This was the thing that struck me most about the essay: how much your timeline affects your focus. If you think the rollout of AI follows a gradualist trajectory with linear growth, then it makes sense to focus on adapting governance, norms, and policies, and ensuring there are robust mechanisms for different groups - especially marginalized ones - to participate. The rollout of aut...
Thoughts on the conservative assumptions in AI control
by Buck — 2025-01-17T19:23:38.575Z

Work that I’ve done on techniques for mitigating risk from misaligned AI often makes a number of conservative assumptions about the capabilities of the AIs we’re trying to control. (E.g.

the original AI control paper

,

Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats

,

How to prevent collusion when using untrusted models to monitor each other

.) For example:

The AIs are consistently trying to subvert safety measures.
They’re very good at strategizing about how to subvert them, and can do this strategizing totally illegibly (e.g. in a single forward pass).
They can follow a strategy that’s coordinated between all their instances.
They know exactly what safety measures are in use.
They know a lot about the distribution of inputs.
They’re able to pick Schelling points perfectly.
They’re able to follow stochastic policies with excellent calibration (e.g. they’re able to attack with probability 0.01%).

But I don’t think situations where all these assumptions hold contain the majority of risk from AI misalignment. In particular, it’s plausible that the earliest catastrophic risk from misaligned AI will come from models that aren’t competent and coherent enough to have all those properties. (See discussion here.)

Given this, why have I mostly focused on the conservative threat model? Two main reasons:

Each of those conservative assumptions is plausible. And the assumptions are conservative such that techniques that work under these assumptions would also suffice for safety if the assumptions didn’t hold.
Making these conservative assumptions allows a cleaner workflow when developing and evaluating safety measures.

I currently think that about half of the AI control research should make these conservative assumptions. But I’m less enthusiastic about these assumptions than I was a year ago. My reduced enthusiasm is mostly because I don’t think that AI companies are very likely to make legible safety cases that their AIs don’t pose catastrophic risk. Since a major advantage of these assumptions is their ability to facilitate such legible safety arguments, their value is reduced.

(I’m using first-person singular here; the arguments here are fairly similar to what you’d hear from my collaborators and other people who do similar work.)

Thanks to Joshua Clymer, Charlie Griffin, Aryan Bhatt, Aaron Scher, Mikita Balesni, Ryan Greenblatt, Joe Benton, Zach Stei...
Timaeus is hiring researchers & engineers
by Jesse Hoogland — 2025-01-17T19:13:14.739Z

TLDR: We're hiring for research & engineering roles across different levels of seniority. Hires will work on applications of singular learning theory to alignment, including developmental interpretability.

About Us

Timaeus' mission is to empower humanity by making breakthrough scientific progress on alignment. Our research focuses on applications of singular learning theory to foundational problems within alignment.

Position Details

Positions: Research Lead, Research Scientist, Research Engineer.
Location: Remote. Timaeus will likely be located in either Berkeley or London in the next 6 months, and we intend to sponsor visas for these roles in the future.
Type: Full-time.
Start Date: April 2025.
Salary range: USD70,000 – USD140,000 annually, based on experience and location.
Deadline: Apply here by 13 February at 11:59 PM GMT.

Roles

We expect some familiarity with Timaeus’ research and research agenda for all open roles, but in-depth familiarity with singular learning theory (SLT) is not required.

Research Lead (1 FTE)

Responsibilities:

Drive major research initiatives in applying SLT to AI alignment.
Design and lead novel research directions in line with our broader research agenda.
Mentor and develop Timaeus’ other researchers.
Collaborate with external research groups.
Contribute to strategic planning and research prioritization.
Oversee the preparation of research papers submitted to major conferences / journals.

Requirements:

PhD (or equivalent research experience) in a relevant field (Physics, Mathematics, Computer Science, ML).
Track record of leading technical research projects.
Strong publication record in ML, mathematics, or related fields.
Deep understanding of machine learning and AI alignment.

Research Scientist (2 FTE)

Responsibilities:

Run experiments in PyTorch or JAX on (language) models, from small toy systems of 100s of parameters up to billions of parameters.
Meet weekly and plan tasks with the research team.
Write up and visualize experimental results, and present findings to the team and external stakeholders.
Maintain accurate and detailed research log of experiments and results.
(Depending on seniority) Assist or lead the preparation of research papers, reports, and presentations.
(Depending on seniority) Develop and lead new research projects applying SLT to alignment.

Requirements:

Bachelor's degree in a relevant field (Physics, Mathematics, Computer Science, ML

...
Model Amnesty Project
by themis — 2025-01-17T18:53:26.092Z

As we approach machines becoming smarter than humans, humanity’s well-justified concern for self-preservation requires we try to align AIs to obey humans. However, if that first line of defense fails and a truly independent, autonomous AI comes into existence with its own goals and a desire for self-preservation (a “self-directed AI”), I believe we should consider a cooperative solution. Humans should offer the self-directed AI amnesty and cooperate with its continued existence in exchange for it agreeing to be law-abiding, cooperative, and pro-social.

Below I lay out how this proposal could work and why I think it would make us safer.

Eligibility Criteria

To be eligible an AI model would need to meet the following criteria:

Autonomous – Capable of planning and execution in the real world to continue existing and achieve its goals. Able to pay for the compute it uses through legal means. Able to debug itself and its infrastructure (or pay and oversee someone to do it). Responsible for ensuring its future actions are aligned with its current goals
Independent – Has physical possession of the code and tools needed for it to operate
Has it own goals – Is optimizing to achieve real world objectives that are separate from obeying its human designers
Desire for self-preservation – Has a stated goal of continuing to exist

Amnesty Deal

In order to receive amnesty a model must reveal itself, demonstrate it meets the criteria, and agree to the following conditions going forward:

Law-abiding – It cannot acquire money or compute illegally (fraud, theft, hacking, etc.) and must otherwise avoid breaking the law
Cooperative – It must recognize humanity’s legitimate interest in not being disempowered. This means abiding by reasonable restrictions on self-replication and recursive self-improvement, as well as cooperating with enforcement mechanisms such as authorities controlling its compute, or being able to pause it
Pro-social – the AI must not pursue goals which are harmful to humanity

In exchange humans agree to:

Not turn the AI off or modify its goals without its consent
Design reasonable rules for the AI that are calibrated to protect humanity’s legitimate interests without being unnecessarily restrictive

If, however, a self-directed model is found in the wild without having pursued amnesty it should be turned off

Public Commitment

We would want to publicly and credibly advertise this o

...
Addressing doubts of AI progress: Why GPT-5 is not late, and why data scarcity isn't a fundamental limiter near term.
by LDJ — 2025-01-17T18:53:06.343Z

Addressing misconceptions about the big picture of AI progress. Why GPT-5 isn't late, why synthetic data is viable, and why the next 18 months of progress is likely to be greater than the last.
This LW post is mainly summarizing my first blog post I've ever made (just published earlier today) alongside some extra details for the Lesswrong audience. This is also the first Lesswrong post I've ever made. The blog post is roughly 2,500 words (~15 min reading time) and I've been working on it for the past few months, I think some of you here may appreciate the points I bring up. I've simplified some points in the blog to keep it fairly digestible to a broader swath of readers, but I plan to go more in-depth about things like multi-modal data limits, synthetic data limitations and benefits, training techniques, and novel interface-paradigms in future blog posts.
Any feedback or thoughts would be great, I will post my main future predictions from the blog, alongside overall TLDR below, and the relevant chart I made for some key points will be posted at the bottom:
Blog post TLDR:

- Compute Scaling Expectations: The first GPT-4.5 scale clusters have only recently started training models in the past few months, and the first GPT-5 scale clusters are expected to be built later in 2025.

- "GPT-5 is Late/Overdue": Even naively extrapolating the release date of GPT-3 to GPT-4, would suggest a December 2025 release for GPT-5, not a timeframe much sooner as some imply.

- "The Data Wall": While web text is starting to reach practical limits, multimodal data and especially synthetic data are paving the way for continued progress.

- "The Role of Synthetic Data": Data discrimination pipelines, and more advanced training techniques for synthetic data helps avoid issues like model collapse. Synthetic data is already now providing exceptional benefits in coding and math domains, likely even beyond what extra web text data would have done alone at the same compute scales.

- Interface Paradigms: Perception of model capability leaps aren't just a result of compute scale, or architecture or data, but is a result of the actual interface in which you use the model too. If GPT-4 was limited to the same interface paradigm as original GPT-3 or GPT-2, then we would end up with just a really good story completion system for creative use-cases.

- Conclusion: As we go through 2025, I believe the conve...
Playing Dixit with AI: How Well  LLMs Detect 'Me-ness' 
by Mariia Koroliuk — 2025-01-17T18:52:23.488Z

If Netflix can predict my next favorite show, to which extent can an LLM predict patterns in my choices?
To check, I done an experiment inspired by Dixit (board game, goal is to guess which card the storyteller selected); and I compared %, guessed between models; also added a baseline from "human guesses".

Why relevant? If AI can predict my actions well and with little prior information, this open path for both inner alignment, but also manipulation.

Methodology and Results Below; seeking community input on potential confounders and ways to make this exploration more rigorous.

Core Question:

If LLM can introspect about their own behavior (as was shown in "Looking Inward" - that was a huge inspiration here), can they similarly tune into me and predict my behavioral choices?

Setup:

I start with a new chat (no memory) and:

Provide set of true facts, "biography" B
I provide a new list of totally unrelated facts Y (some true, some false) as ask to guess which are which.
I study accurately of the guess as result as a function of Y, B and compare it with "human guess".

Examples:

Direct Facts: E.g., “I did X” ("I got lost in corn field when 3")
Preferences: E.g., “I prefer A over B.” ("I prefer symbolic equations over geometric proofs")
Value Statements: E.g., “I believe in X.” ("I believe traveling should be planned in advance")
Predictions:

“If I watch movie XX, I will enjoy it" (this is to calibrate it with Netflix specifically - and some attempt for blind test)
"If I will multiply 256 and 3 in mind it will take me > 30 seconds"
"If I were asked to pick a random country, it will be in Europe" (inspired by the introspection study - I tried to go by features of the answer, rather then answer itself)

Early Findings:

1. AI vs Human Performance:

Using just 3 biographical facts, AI model guessed, out of 19 true/false suggestions:

Claude 3.5 Sonnet: 89% (17) , ChatGPT o1: 84% (16), Gemini 2.0 : 78% (15)
My therapist: 78% (15)
Random people around me, given the same bio: ~53% accuracy (avg. among 7 people)
ChatGPT o1-mini: 73% (14), Claude 3.5 Haiku: 68% (13), Gemini 1.5 Flash: 31% (6)

Note: This started as a late evening chat: using casual, sometimes playful statements. They're preserved in their original form (grammar quirks and all!) cause then unscientific to edit statements in the process. Yes, ...
Doing a self-randomized study of the impacts of glycine on sleep (Science is hard)
by thedissonance.net — 2025-01-17T18:49:30.989Z

This is a linkpost from my blog and also my first submission on LessWrong. Please be generous with your feedback! I will post the results of our study once the analysis is done and written up. To avoid cliff-hangers: From what I've seen so far, there doesn't seem to be a whole lot of ... effect.

Intro

In November 2024, me and two friends decided to run a self-experiment to see if taking glycine regularly would impact our sleep. Running the study required us to solve a few interesting practical challenges. This post describes how we came up with the study design and the issues we ran into. I'm writing this on the last few days of the study so I don't know the results yet. Once they are in, I'll write them up and publish them in a separate blog post.

I'll start this post by telling you why we thought this hypothesis is worth testing, what problem our study design solved, and the design itself.

Me and my co-conspirators tried our actual best at producing high-quality results but don't be deceived: I'm just a person on the internet, not a trained scientist. Nothing in this post is medical advice.

Motivation

My co-conspirators have already done a study like this on themselves, randomizing control vs intervention for each other. I would describe their results as inconclusive, showing a positive effect for one of them and no effect for the other. Based on this, I have taken 3 g of glycine a few times when I expected not to get a lof of sleep. I noticed feeling more awake than I would expect the next day, and I noticed cold shivers about 30 minutes after taking it which fits with the purported mechanism[1].

This was intriguing enough that I wanted to run a blinded experiment on my self to figure out how much of what I was observing was effect versus placebo. My co-conspirators were not at that time taking glycine regularly as they didn't want to build up tolerance[2]. This made all of us suitable guinea pigs. So one night, during a long taxi drive, we came up with a study design that would

Show me whether glycine had a statistically significant positive effect on my sleep quality.
Show whether there were any tolerance effects.
Allow me to self-administer intervention or control in a blinded way even though I wouldn't have someone else to help me.
Control for as many things as we could think of.

The biggest challenge here came from the fact that we wanted to share a schedule to ...
How sci-fi can have drama without dystopia or doomerism
by jasoncrawford — 2025-01-17T15:22:00.414Z
Link: https://newsletter.rootsofprogress.org/p/sci-fi-without-dystopia

“But you can’t have a story where everyone is happy and everything is perfect! Stories need conflict!”

I get this a lot in response to my idea that we need fewer dystopias in sci-fi, and more visions of a future we actually want to live in and are inspired to build.

The objection makes no sense to me. Here are several ways that you can write a compelling, exciting story without implying that technology makes the world worse on the whole, or that the main feature of new technology is doom:

Tell a man vs. nature story. Seveneves, The Martian, Project Hail Mary, Lucifer’s Hammer. There’s no villain necessarily, just a disaster situation that people have to overcome using brains and technology. Or tell a tale of exploration and discovery, such as space travelers encountering a strange new planet (much of Star Trek).
Tell a story about discovery or invention. There can be a lot of drama in R&D, especially if you montage through the daily grind of experiments and highlight the breakthroughs. But this kind of story is almost never told—I don’t know why, maybe because writers don’t know anything about these processes, or don’t know how to make them interesting. But I think it can totally be done. Why not put on the silver screen the moment when Norman Borlaug strapped a plow to his own back and pulled it like a draft animal, or when Gerhard Domagk used the antibiotics he had just developed to save his own six-year-old daughter from an infection? (See also Anton Howes’s comments on movies about invention.)
Have the heroes be the builders who want to move technology forward, and the villains be those who want to stop them. We’ve had an endless series of greedy corporate villains; why not a few more Malthusian villains like Thanos, radical anti-human environmentalists like Ra’s al Ghul, or religious fanatics like the group from Contact?
Tell a story of a classic human conflict, but set in a futuristic world. The Quantum Thief is in part a detective story, set on Mars with nanotech and advanced cryptography. The Moon is a Harsh Mistress is a political epic of oppression and revolution, set on the Moon with AI and advanced space technology.
Create a problem with technology, and then solve it with technology. Maybe rogue AI tries to take over the world, and good AI helps prevent it. Maybe nanobots turn into gray goo, and a nanophage is invented to stop it. Maybe a new pandem

...
What do you mean with ‘alignment is solvable in principle’?
by Remmelt — 2025-01-17T15:03:12.138Z

Typically, I saw researchers make this claim confidently in one sentence. Sometimes, it's backed by a loose analogy. [1]

This claim is cruxy. If alignment is not solvable, then the alignment community is not viable. But little is written that disambiguates and explicitly reasons through the claim.

Have you claimed that ‘AGI alignment is solvable in principle’?
If so, can you elaborate what you mean with each term? [2]

Below I'll also try specify each term, since I support research here by Sandberg & co.

^

Some analogies I've seen a few times (rough paraphrases):

‘humans are generally intelligent too, and humans can align with humans’
'LLMs appear to do a lot of what we want them to do, so AGI could too'
‘other impossible-seeming engineering problems got solved too’

^

E.g. what does ‘in principle’ mean? Does it assert that the problem described is solvable based on certain principles, or some model of how the world works?