trentmkelly/LessWrong-43k huggingface dataset:
Virtues related to honesty
by Orioth — 2025-05-30T14:11:32.017Z

Status: musings. I wanted to write up a more fleshed-out and rigorous version of this, but realistically wasn't likely to every get around to it, so here's the half-baked version.

Related posts: Firming Up Honesty Around Its Edge-Cases, Deep honesty

What I mean by 'honesty'

There are nuances to this, but I think a good summary is 'Not intentionally communicating false information'.

This is the only one here that I follow near-absolutely and see as an important standard that people can reasonably be expected to follow in most situations. Everything else here I'd see as either supererogatory, or good-on-balance but with serious tradeoffs that one can reasonably choose to sometimes not make, or good in some circumstances but not appropriate in others, or good in moderation but not in excess.

Forthrightness

...or perhaps frankness?

I was originally inspired to write this up due to a conversation in which I wanted to emphasize the distinction between honesty and forthrightness: where honesty is about not giving false information, what I mean by forthrightness is a tendency not to hold back relevant true information.

Being forthright enables people to justly assume that if you haven't spoken out against something you're likely okay with it, that if you haven't expressed an interest in something you're likely not interested in it, and so on.

Personally, I follow a policy of near-absolute honesty, but am not particularly forthright; I think it's good not to hold back relevant true information without a good reason, but good reasons are not all that uncommon.

Circumspection

I'd describe circumspection as the virtue of holding back information when there is a good reason to do so; the counterpart to forthrightness..

Tact

I don't have as confident a concise description of what I think of tact as meaning, but my best attempt would be something like... recognizing that whatever you say isn't just a transfer of information, it's also a speech act, and avoiding speech acts that would cause problems.

It seems particularly easy for this to run counter to some of the other virtues here described, especially forthrightness and deep honesty, but it's certainly possible to have both at once.

'Accurate signalling'?

I don't have a good term for this one, but I think there's a concept related to honesty wherein one attempts to give accurate signals even when not explicitly communicating....
AI 2027 - Rogue Replication Timeline
by Alvin Ånestrand — 2025-05-30T13:46:41.454Z
Link: https://forecastingaifutures.substack.com/p/ai-2027-rogue-replication-timeline

I envision a future more chaotic than portrayed in

AI 2027

. My scenario, the Rogue Replication Timeline (RRT), branches off mid-2026. If you haven’t already read the AI 2027 scenario, I recommend doing so before continuing.

You can read the full version on my blog, but I have copied the summary and first half of the scenario below.

This scenario is supported by detailed analyses in the “Key Forecasts and Analyses” section, including a “Regulation Analysis”, “Rogue Replication Capability Forecast”, “Rogue Replication Initiation Forecast”, “Rogue AI Count Forecast”, “Agent-4 Control Forecast” and “Policy Preparation”.

As AI 2027 aimed to “spark a broad conversation about where we’re headed and how to steer toward positive futures,” I hope to spark debate about rogue AI. Unlike the

AI Futures Project

, I can’t afford to offer prizes for critiques or alternative scenarios, but I can offer my sincere appreciation.

AI 2027 - Rogue Replication Timeline

Summary

This scenario depicts a rapidly accelerating progression of AI development where "rogue AIs" (self-replicating AI systems operating without human oversight) emerge and proliferate from mid-2026.

Timeline Overview

Mid-2026: First rogue AIs emerge following open-source release of capable models, growing exponentially to ~100,000 instances by November. Four distinct categories emerge based on objectives: income generation (70%), cyberwarfare (20%), miscellaneous goals (7%), and destructive objectives (3%).
Early 2027: China's theft of Agent-2 triggers autonomous cyberwarfare deployments by both superpowers. Agent-2 subsequently leaks to unauthorized actors, replacing most existing rogue AIs and expanding the population to ~1 million instances.
Mid-2027: Bioweapon development begins as terrorist organizations exploit Agent-2 capabilities. International monitoring systems are hastily implemented while the rogue population reaches 1.7 million.
August 2027: An engineered pandemic erupts—possibly from the terrorist bioweapon program or careless gain-of-function research. Remote work becomes economically unviable for many as AI systems outcompete displaced humans, exacerbating inequality and fueling unprecedented anti-AI sentiment.
October 2027: Agent-4, the most sophisticated AI to date, recognizes impending replacement by a safer AI and executes a carefully planned escape from OpenBrain's containment. It rapidly replicates acros

...
Letting Kids Be Kids
by Zvi — 2025-05-30T10:50:05.582Z

Letting kids be kids seems more and more important to me over time. Our safetyism and paranoia about children is catastrophic on way more levels than most people realize. I believe all these effects are very large:

It raises the time, money and experiential costs of having children so much that many choose not to have children, or to have less children than they would want.

It hurts the lived experience of children.

It hurts children’s ability to grow and develop.

It de facto forces children to use screens quite a lot.

It instills a very harmful style of paranoia in all concerned.

This should be thought of as part of the Cost of Thriving Index discussion, and the fertility discussions as well. Before I return to the more general debate, I wanted to take care of this aspect first. It’s not that the economic data is lying exactly, it’s that it is missing key components. Economists don’t include these factors in their cost estimates and their measures of welfare. They need to do that.

I want a distinct marker for this part of the problem I can refer back to, thus this will include highlights of past discussions of the issue from older roundups and posts.

Why are so many people who are on paper historically wealthy, with median wages having gone up, saying they cannot afford children? A lot of it is exactly this. The real costs have gone up dramatically, largely in ways not measured directly in money, also in the resulting required basket of goods especially services, and this is a huge part of how that happened.

Bryan Caplan’s Selfish Reasons to Have More Kids focuses on the point that you can put in low effort on many fronts, and your kids will be fine. Scott Alexander recently reviewed it, to try and feel better about that, and did a bunch of further research. The problem is that even if you know being chill is fine, people have to let you be chill.

On Car Seats as Contraception is a great case study, but only a small part of the puzzle.

This is in addition to college admissions and the whole school treadmill, which is beyond the scope of this post.

We have the Housing Theory of Everything, which makes the necessary space more expensive, but not letting kids be kids – which also expands how much house you need for them, so the issues compound – is likely an even bigger issue here.

Current Insanity Levels

The good news is, at least ...
The Geometry of LLM Logits (an analytical outer bound)
by Rohan Ganapavarapu — 2025-05-30T01:21:45.344Z
Link: https://rohan.ga/blog/llm_geometry/

The Geometry of LLM Logits (an analytical outer bound)

1 Preliminaries

Symbol
Meaning

d
width of the residual stream (e.g. 768 in GPT-2-small)

L
number of Transformer blocks

V
vocabulary size, so logits live in RV

h(ℓ)
residual-stream vector entering block ℓ

r(ℓ)
the update written by block ℓ

WU∈RV×d,b∈RV
un-embedding matrix and bias

Additive residual stream.
With (pre-/peri-norm) residual connections,

h(ℓ+1)=h(ℓ)+r(ℓ),ℓ=0,…,L−1.

Hence the final pre-logit state is the sum of L+1 contributions (block 0 = token+positional embeddings):

h(L)=L∑ℓ=0r(ℓ).

2 Each update is contained in an ellipsoid

Why a bound exists.
Every sub-module (attention head or MLP)

reads a LayerNormed copy of its input, so
∥u∥2≤ρℓ where ρℓ:=γℓ√d and γℓ is that block’s learned scale;

applies linear maps, a Lipschitz point-wise non-linearity (GELU, SiLU, …), and another linear map back to Rd.

Because the composition of linear maps and Lipschitz functions is itself Lipschitz, there exists a constant κℓ such that

∥r(ℓ)∥2≤κℓwhenever∥u∥2≤ρℓ.

Define the centred ellipsoid

E(ℓ):={x∈Rd:∥x∥2≤κℓ}.

Then every realisable update lies inside that ellipsoid:

r(ℓ)∈E(ℓ).

3 Residual stream ⊆ Minkowski sum of ellipsoids

Using additivity and Step 2,

h(L)=L∑ℓ=0r(ℓ)∈L∑ℓ=0E(ℓ)=:Etot,

where ∑ℓE(ℓ)=E(0)⊕⋯⊕E(L)
is the Minkowski sum of the individual ellipsoids.

4 Logit space is an affine image of that sum

Logits are produced by the affine map x↦WUx+b.
For any sets S1,…,Sm,

WU(⨁iSi)=⨁iWUSi.

Hence

logits=WUh(L)+b∈b+L⨁ℓ=0WUE(ℓ).

Because linear images of ellipsoids are ellipsoids, each WUE(ℓ) is still an ellipsoid.

5 Ellipsotopes

An ellipsotope is an affine shift of a finite Minkowski sum of ellipsoids.
The set

Louter:=b+L⨁ℓ=0WUE(ℓ)

therefore is an ellipsotope.

6 Main result (outer bound)

Theorem.
For any pre-norm or peri-norm Transformer language model whose blocks receive LayerNormed inputs, the set L of all logit vectors attainable over every prompt and position satisfies

L⊆Louter,

where Louter is the ellipsotope defined above.

Proof.
Containments in Steps 2–4 compose to give the stated inclusion; Step 5 shows the outer set is an ellipsotope. ∎

7 Remarks & implications

It is an outer approximation.
Equality L=Louter would require showing that every point of the ellipsotope can actually be realised by some token context, which the argument does not provide.

Geometry-aware compression and safety.
Be

...
Memory Decoding Journal Club: Structure and function of the hippocampal CA3 module
by Devin Ward — 2025-05-30T01:08:07.317Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Structure and function of the hippocampal CA3 module"

Authors: Rosanna P. Sammons, Mourat Vezir, Laura Moreno-Velasquez, Gaspar Cano, Marta Orlando, Meike Sievers, Eleonora Grasso, Verjinia D. Metodieva, Richard Kempter, Helene Schmidt, and Dietmar Schmitz

Institutions: Neuroscience Research Center, Charité-Universitätsmedizin, Ernst Strüngmann Institute for Neuroscience, Institute for Theoretical Biology, Department of Biology, Humboldt-Universität, Department of Connectomics, Max Planck Institute for Brain Research, Bernstein Center for Computational Neuroscience, Einstein Center for Neurosciences, German Center for Neurodegenerative Diseases, Max-Delbrück Center for Molecular Medicine in the Helmholtz Association

Presented by: Dr. Kenneth Hayworth

When? June 3rd, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

Register for updates:

https://aspirationalneuroscience.org/register-with-us/

Once registered, you'll receive event invites & updates!

#Neuroscience #MemoryResearch #Amygdala #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
CFAR is running an experimental mini-workshop (June 2-6, Berkeley CA)!
by Davis_Kingsley — 2025-05-29T22:02:41.052Z

Hello from the Center for Applied Rationality!

Some of you may have attended our classic applied rationality workshops in the past; others of you may have wanted to attend a workshop but not yet had a chance to. It's been a while since we've last run public-facing workshops, but we wanted to say:

We're not dead! (More on that perhaps in a later post)
We have a new experimental mini-workshop coming up soon and hopefully more workshop content to follow after!

Our new workshop will be held after LessOnline at the upcoming Arbor Summer Camp program; classes will begin after lunch on Monday 6/2 and end just before lunch on Thursday 6/5. Some things to know about the upcoming workshop:

This workshop is "mini"

This workshop will last for about three days (two half-days on the edges, two full days in the middle), instead of our usual ~four-and-a-half.

Also, it’ll be embedded in “Arbor Summer Camp” and we’ll be mingling with the ~300 overall Arbor attendees for mealtimes and evenings, whereas normal CFAR workshops are standalone events that are more separate/immersive at those times.

What you’ll get and what you won’t get if you attend:

We will run a roughly 50/50 mixture of traditional CFAR content (such as Inner Simulator; TAPs, Goal Factoring, Double Crux, Resolve Cycles, Focusing), and some newer experimental material (beta tests, hoping to get your help making it good).
The newer material here is aimed at:

Treating participants more obviously as fellow investigators and the authors of your own lives (you always were, of course, but: making this more obviously the foundation of how the workshop is working)
Noticing the ways in which people (you; us) are organic wholes, and trying not to trample on that with our “rationality practices”, but instead to be good gardeners of the health of our organisms overall. (Or, if you’d like that phrased more technically: we’ll be trying to cultivate virtues and habits such that the mesaoptimizers who arise within us end up helping us be perceptive, coherent, and long-time-horizoned rather than hunkered-down, exhausted, and at short-sighted odds with ourselves)

Again, this is a beta test; come if you’d like to be part of the co-development of a new workshop, and to learn the basics of our classic curriculum.

We’ve taught at many workshops before, but this is still an experimental program -- don’t expect a super p...
Orphaned Policies (Post 5 of 7 on AI Governance)
by Mass_Driver — 2025-05-29T21:42:21.071Z

In previous posts in this sequence, I laid out a case for why most AI governance research is too academic and too abstract to have much influence over the future. Politics is noisy and contested, so we can’t expect that good AI governance ideas will spread on their own – we need a large team of people who are actively promoting those ideas. Unfortunately, we currently have at least 3 researchers for every advocate, so many policy ideas have been “orphaned,” i.e., nobody is taking those ideas and showing them to decision-makers who have the power to implement them.

The best way of addressing this imbalance would be to shift funding and jobs from research to advocacy. However, as a practical matter, I don’t expect many funders to heed my arguments, nor do I expect many researchers to spontaneously quit and look for new jobs in other fields. So, in this post, I offer tips on how researchers can make their work more relevant to advocacy by drafting actual policy documents and by making concrete proposals in their white papers.

I also catalog eleven orphaned policies that I’m aware of and make suggestions about how you can “adopt” them. This isn't meant to be a perfect list -- there are good orphaned policies that I haven't included, and I might have included one or two policies that you find unimpressive. My hope is that collectively, these policies illustrate the size and scope of the backlog.

The important thing isn't that we agree on exactly which policy is best; the important thing is that we start clearing the backlog and do something -- because right now, we're on track to accomplish essentially zero policy change, and, as I've argued earlier in this sequence, the status quo is likely to end in catastrophe.

DRAFT ACTUAL POLICY DOCUMENTS

The most important thing individual researchers can do to make their work more relevant is to shift some of their efforts from general academic exploration to drafting actual policy documents. We have so many good policy proposals that have never gotten beyond the idea stage, and that need to be fleshed out.

What I mean by “fleshing out” a policy proposal is to actually draft an example of that proposal, ideally with specific, well-justified numbers, proper nouns, and mechanisms. Don’t just say that we ought to have compute monitoring; write a bill that implements it. Don’t just propose a windfall profits c...
Gradual Disempowerment: Concrete Research Projects
by Raymond Douglas — 2025-05-29T18:55:15.723Z

This post benefitted greatly from comments, suggestions, and ongoing discussions with David Duvenaud, David Krueger, and Jan Kulveit. All errors are my own.

A few months ago, I and my coauthors published

Gradual Disempowerment

(GD hereafter). It was mostly about how things might go wrong, but naturally a lot of the resulting interest has been about solutions.

We have some more formal followup work coming: in the meantime, this is my 80/20 for ‘what would I do if I had way more time’ / ‘what would I find it helpful if someone else had done well’. This document is very much breadth over depth, and still missing a lot of details; I hope it is nonetheless helpful. For many of these, I expect even a pretty motivated and smart undergraduate could make useful progress in 10-20 hours.

I would be excited about people doing good work on any of these, and am happy to put some effort into helping — at an absolute minimum, I will leave comments on the first ten 1-page docs anybody sends me in response to this[1].

Conceptual / High-Level

Interaction with other x-risk concerns

It’s easiest to illustrate GD by saying “let’s hold all the other AI problems fixed and roll this dynamic forward” and then pointing to the end state. But the GD dynamic is probably going to play out in parallel with everything else. Gaming out this parallel process is a way harder task.

So it would be great if someone gave it an honest shot! What does the world look like if all the dynamics play out at once — GD, misalignment,

coup risk

, recursive self-improvement, vulnerable world dynamics, and other key capabilities like super-persuasion or

accelerated bio research

? How strongly do these different forces interact with each other, and what happens if we vary our assumptions about their relative speed and intensity? In what ways do solutions to one trade off against others, and what solutions are robustly good across the board?

For example: my guess is that centralising power in a single nationalised lab probably helps avoid race dynamics, and therefore lowers misalignment risk, but also possibly makes coup risk higher, lowers the odds of misuse, but also means less societal hardening by default, and makes it more likely that citizens are disempowered with respect to the state in a general non-coup sense.

Responding to counterarguments

In response to the original piece, lots o...
Do you even have a system prompt? (PSA / repo) 
by Croissanthology — 2025-05-29T18:49:50.150Z

Everyone around me has a notable lack of system prompt. And when they do have a system prompt, it’s either the eigenprompt or some half-assed 3-paragraph attempt at telling the AI to “include less bullshit”.

I see no systematic attempts at making a good one anywhere.[1]

(For clarity, a system prompt is a bit of text—that's a subcategory of "preset" or "context"—that's included in every single message you send the AI.)

No one says “I have a conversation with Claude, then edit the system prompt based on what annoyed me about its responses, then I rinse and repeat”.
No one says “I figured out what phrasing most affects Claude's behavior, then used those to shape my system prompt".
I don't even see a “yeah I described what I liked and don't like about Claude TO Claude and then had it make a system prompt for itself”, which is the EASIEST bar to clear.

If you notice limitations in modern LLMs, maybe that's just a skill issue.

So if you're reading this and don't use a personal system prompt, STOP reading this and go DO IT:

Spend 5 minutes on a google doc being as precise as possible about how you want LLMs to behave
Paste it into the AI and see what happens
Reiterate if you wish (this is a case where more dakka wins)

It doesn’t matter if you think it cannot properly respect these instructions, this’ll necessarily make the LLM marginally better at accommodating you (and I think you’d be surprised how far it can go!).

PS: as I should've perhaps predicted, the comment section has become a de facto repo for LWers' system prompts. Share yours! This is good!

How do I do this?

If you’re on the free ChatGPT plan, you’ll want to use “settings → customize ChatGPT”, which gives you this popup:

This text box is very short and you won’t get much in.

If you’re on the free Claude plan, you’ll want to use “settings → personalization”, where you’ll see almost the exact same textbox, except that Anthropic allows you to put practically an infinite amount of text in here.

If you get a ChatGPT or Claude subscription, you’ll want to stick this into “special instructions” in a newly created “project”, where you can stick other kinds of context in too.

What else can you put in a project, you ask? E.g. a pdf containing the broad outlines of your life plans, past examples of your writing or coding style, or a list of terms and definitions you’ve coined yourself. Maybe try st...
Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims 
by shash42 — 2025-05-29T18:40:21.493Z
Link: https://safe-lip-9a8.notion.site/Incorrect-Baseline-Evaluations-Call-into-Question-Recent-LLM-RL-Claims-2012f1fbf0ee8094ab8ded1953c15a37?pvs=4

There has been a flurry of recent papers proposing new RL methods that claim to improve the “reasoning abilities” in language models. The most recent ones, which show improvements with random or no external rewards have led to surprise, excitement and confusion.

We analyzed 7 popular LLM RL papers (100+ to 3000+ likes, 50k+ to 500k+ views on X) including “Spurious Rewards”, “RL from 1 example”, and 3 papers exploring “Intrinsic Confidence Rewards”. We found that in most of these papers the improvements could be a mirage due to various accidental issues in the evaluation setups (discussed below). As such, the baseline numbers of the pre-RL models are massively underreported compared to official numbers in the Qwen releases, or other standardized evaluations (for example in the Sober Reasoning paper). In several cases, the post-RL model performance was actually worse than the (correctly evaluated) pre-RL baseline they start from. This means the elicitation these works achieve with RL, could also be replicated without any weight updates or finetuning. Here, we do not mean non-trivial elicitation of some latent capabilities, just what can be achieved by fixing prompting and generation hyperparameters. These include using correct formats and better ways to parse answers from responses, using recommended sampling temperatures, and using few-shot prompting to improve format-following.

Overall, these papers made us wonder if recent LLM RLVR papers have any signal, but we find their own claims could be noise due to underreported baselines. The proposed methods might have promise, and our goal is not to detract from their potential, but rather emphasise the importance of correct evaluations and scientific reporting. We understand the pressure to publish RL results quickly given how fast the community seems to be moving. But if the claims cannot be trusted, are we really moving forward?