trentmkelly/LessWrong-43k huggingface dataset:
What life will be like for humans if aligned ASI is created
by james oofou — 2025-03-27T10:06:56.846Z

Let's assume computationalism and the feasibility of brain scanning and mind upload. And let's suppose one is a person with a large compute budget.

In this post I'll attempt to answer these quetions: How should one spend one's compute budget? How many uploads of oneself should one create? Should one terminate one's biological self? What will one's uploaded existence be like?

First, let's establish the correct frame in which to explore the questions relating to the act of upload. One is considering creating copies of oneself. So, what happens, subjectively, when one spins up a new copy of oneself? The copy is computationally identical to the original, and consciousness is computation, so each is equally oneself. But one is not both. This means that when one is creating a copy one can treat it as a gamble: there's a 50% chance they find themselves in each of the continuations.

What matters to one is then the average quality of one's continuations. One does not benefit from the creation of many great continuations over just one great continuation, because one can never experience more than one continuation.

Therefore, one should spend all of one's resources on creating the single best possible continuation. And one should terminate one's biological self: the real world is strictly worse than the ASI-curated personal utopia one's upload will experience, so terminating the biological self increases the average quality of one's continuations. To make this point clear: if one did not terminate the biological self, one would subjectively have a 50% chance of ending up in the less-favourable biological psychological continuation.

But we can figure out more about this continuation. One wants to make sure one is spending one's compute selfishly (to whatever extent one is altruistic, one should allocate a proportionate amount of one's compute budget to ASI). But value drift is a danger here: one's life expectancy is going to be absurdly high, maybe undecillions of years of personal utopia, so by default one's beliefs will evolve, and one's pre-ASI memories will be forgotten... every aspect of oneself will evolve.

Before even a small fraction of one's life has played out, one's copy will bear no relation to oneself. To spend one's compute on this person, effectively a stranger, is just altruism. One would be better off donating the compute to ASI. ...
What is scaffolding?
by Vishakha — 2025-03-27T09:06:35.403Z
Link: https://aisafety.info/questions/NM25/What-is-scaffolding

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

"Scaffolding" is a fuzzy term referring to code built up around an LLM in order to augment its capabilities[1]. This does not typically include code which alters the LLM's internals, such as fine-tuning or activation steering.[2]

People use scaffolding because it can allow LLMs to use tools, reduce error rates, search for info, etc., all of which make LLMs more capable.

Scaffolding is an important aspect of safety evaluations because, once an LLM is deployed, users will inevitably attempt to use it to make the LLM more powerful. Dedicated effort should be put into pushing the functionality of the LLM to its limits during these evaluations, as the latent capabilities of LLMs are a) what's relevant for dangers and b) hard to tease out with prompts alone.

Another reason to use scaffolding is interpretability. Most of an LLM's reasoning is done within the neural network itself in streams of gigantic tensors which humans don’t yet know how to interpret. But reasoning done via the scaffold is typically done in plain sight via text or code, which is much easier to inspect. E.g., two LLMs communicating with one another to solve a problem might use English to express their reasoning to each other.

Common types of scaffolds include:

Prompt templates: Perhaps the simplest scaffolds, “prompt templates” are just prompts with some blank sections to be filled in at runtime using some local data. E.g., a template like “You are a good assistant. The date is {INSERT DATE HERE}. Tell me how many days are left in the month.” can be filled in with the current date before being used to prompt an LLM.
Retrieval Augmented Generation (RAG): For tasks which have some associated data, RAG is a way of automatically adding task-relevant info to prompts. RAG does this by looking for snippets of text/data in the database which are relevant to the user's prompt, and then adding those snippets to the prompt.
Search engines: Similar to RAG, giving an LLM access to a search engine lets it find info relevant to its prompt. Unlike RAG, the LLM decides what to search for and when.
Agent scaffolds: Scaffolds which try to make LLMs into go

...
Workflow vs interface vs implementation
by Sniffnoy — 2025-03-27T07:38:49.109Z

(Found this old draft that was more or less complete; figured I should publish it. It's adapted from this old blog post of mine.)

It's commonly appreciated that when explaining how to use something, there's a distinction between the high-level usage instructions and the low-level details. But what I've noticed is that different people draw the line a bit differently, and this leads to some confusion. A more useful, less ambiguous distinction, I find, is to break things into three levels, which I'm calling "implementation", "interface", and "workflow".

Implementation is what it sounds like -- how the thing works under the hood, ignoring all abstractions you might use to paper it over. You normally shouldn't have to worry about this unless something breaks, or you want to modify it, or maybe you just want to learn how it works so you can do something new, or whatever. It's what's below the abstraction.

Interface is what the thing does and not how it does it; this is necessarily an abstraction and won't always work, but it's at least what the thing is supposed to do in every case.

Workflow is how you normally use the thing -- not everything it does, just how it will respond when you do the things you normally do with it, not how it's going to respond in weird edge cases.

Note that these distinctions will only really work for artificial things -- something like, say, cooking, doesn't have a clean separation between implementation and interface, and even for something artificial this may not quite work if it's not sufficiently complete or well-implemented -- but this can still be a useful separation to draw even when that separation may not match the territory cleanly.

Now this may sound fairly obvious, but I bother pointing it out because apparently it's not so obvious! I can think of multiple conversations I've had where (in effect) I was given workflow information, found it insufficient, asked about the interface, and was asked, why do you need to know about such internals? Except, I wasn't asking about internals (implementation), I was asking about interface; I didn't want to know how the thing worked interally, just what it would do in cases outside the usual ones. So I think this three-level breakdown, even if it doesn't always map cleanly onto reality, still provides some useful terminology.
Quick thoughts on the difficulty of widely conveying a non-stereotyped position
by Sniffnoy — 2025-03-27T07:30:41.353Z

(This is probably covering subjects already widely discussed here (haven't read LW much recently), but I wanted to get this down quickly and this seemed an appropriate place to post it.)

Here's a problem to which I don't have a solution. How do you get large numbers of people past "valleys of bad rationality" -- conveying a position other than the one they might naturally jump to?

The relevance here is to things like taking ideas seriously, which, as has been often discussed (here's a good recent post by Ozy, or here's perhaps the canonical old one), can be a bit dangerous.

This seems to be pretty hard! Raemon famously stated that you get about five words. Fortunately, I think depending on what you want, it is possible to do a bit better than this. Like, that's if you want to coordinate large numbers of people; if you don't need them to coordinate per se, I think things aren't quite that dire. Eliezer wrote the sequences, those are pretty long, but they're pretty good and a lot of us have read them!

But, there are plenty of people who have come into the rationality community (online or IRL) and have got the "5 words" version of things. A lot of people dislike LW because they don't actually know what it says and are just applying stereotypes... but it's also true that a lot of people like LW for those same mistaken reasons, which can be a pain when they show up and start arguing based on such ideas.

(Here in New York at least we seem to be doing pretty well on this front, in that we've got enough people who've actually taken the time to understand things that when someone shows up at a meetup with the stereotyped understanding we can correct them (as I've seen occur); at least, we can if they state it. But also, it does occur, rather than being completely unnecessary. I don't know how things are elsewhere!)

Note, by the way, that I say "non-stereotyped position", rather than "nuanced position". Nuanced positions are certainly difficult to convey! But the problem is even more general -- even a simple, non-nuanced, extreme positions can sometimes be difficult to get through to people if it it isn't what they're expecting (see: rounding to the nearest cliche). That said, conveying nuanced positions is a big part of the problem, so some of this will focus on that.

So why is this so hard? Well, there are a lot of reasons, but there's a few I want to discuss.

O...
Doing principle-of-charity better
by Sniffnoy — 2025-03-27T05:19:52.195Z

(Thought I'd try posting here some various rationality "quick thoughts" that I feel like I haven't seen expressed overtly enough -- sometimes I've already written about these elsewhere, but thought would be good to get them down here. Not claiming any originality here.)

I think the principle of charity is frequently used in ways that are unhelpful, and I want to make some suggestions on doing principle-of-charity better.

The first thing note I would make is to never apply the principle of charity silently. That is to say, if someone says A, and A seems to you like someone could not possibly assert in good faith, don't just act as if they said reasonable variant B instead and reply to B; explicitly say something to the effect of "A doesn't appear to make any sense to me; did you perhaps mean B? If that is what you meant, then..." Otherwise, you're just deliberately inflicting the AB problem on yourself!

This is especially the case because in many cases there's not a single clearly best charitable interpretation. A common case is when someone makes a statement that has two possible interpretations: One interpretation is interesting but stupid, while the other is true but trivial. (Aka: A "deepity", a troll's truism, or (a special case of) motte-and-bailey doctrine.) What's the charitable interpretation here -- assuming your interlocutor wouldn't deliberately say something stupid, or that they wouldn't deliberately say something trivial?

The answer of course is that this is a silly question because you should never invoke the principle of charity silently. Don't assume -- ask! Say explicitly what you think is going on (although uh probably don't call it "stupid", that's not nice) and ask which interpretation is meant. You can also just not wait for the response, and respond to both possibilities in advance. Doing principle-of-charity correctly sometimes involves a fork.

Before we go further it's worth asking the question, what's the point of the principle of charity anyway? Well the answer is that since it isn't truly a designed thing, more of an evolved thing, it doesn't actually have a singular purpose; rather it seems to potentially have multiple. One purpose might be making people confront the strongest version of their interlocutor's ideas, hence steelmanning -- but see "Against Steelmanning".

But often the main point of saying "assume good faith" is...
X as phenomenon vs as policy, Goodhart, and the AB problem
by Sniffnoy — 2025-03-27T04:32:18.848Z

(Thought I'd try posting here some various rationality "quick thoughts" that I feel like I haven't seen expressed overtly enough -- sometimes I've already written about these elsewhere, but thought would be good to get them down here. Not claiming any originality here.)

When arguing with someone who says we need X, it's important to make sure you know whether they're arguing for X as policy or X as phenomenon. (And it's important to make sure you know which you're arguing for!) By "X as policy", I mean, a policy of directly doing X; by "X as phenomenon", I mean, the phenomenon of X regardless of what policy achieves it. I have noticed many policy arguments where two people talk past each other because they are not clear on this distinction.

(And once they do realize it, they get into terminology arguments instead of tabooing and. I'm hoping introducing this terminology might help avoid that.)

Of course, sometimes people haven't carefully thought about what they want and don't actually mean either of these. Sometimes it's actually just the case that X is a feature of their desired outcome -- OK, actually, strictly speaking that's always true, but let's assume we're implicitly talking within some sort of reasonable bounds so that there are cases where we can reasonably say that it's false.

In this case, optimizing for X (again, within reasonable bounds, otherwise it's true in all cases a person can quickly express!) is unlikely to actually satisfy the person, due to Goodhart's law. That X is a feature of the optimal solution does not mean that one can achieve an optimal solution by focusing on X! Always focus on correctness, not on particular features, even if you are sure a correct solution must have that feature (maybe this is reasonable when judging correctness is very difficult? But be careful with this is my point).

This is all also somewhat related to what I've seen called the "AB problem". (Side note: I've seen multiple people refer to it as this, but I can't seem to find the original source. Anybody know where this term came from? It's not the same as the XY problem, although I suppose there is a relation.)

The AB problem is a problem that occurs in arguments where Yvonne argues for A, Xavier believes A=>B, and so Xavier asserts that Yvonne is arguing for B. But Yvonne does not believe A=>B and is not arguing for B. (Frequently the problem...
Consequentialism is for making decisions
by Sniffnoy — 2025-03-27T04:00:07.020Z

(Thought I'd try posting here some various rationality "quick thoughts" that I feel like I haven't seen expressed overtly enough -- sometimes I've already written about these elsewhere, but thought would be good to get them down here. Adapted from this old blog post of mine but not claiming any originality here. Edit: Here is an earlier similar post by Neil.)

There's an argument I've seen a number of times on the internet about the failings of consequentialism as a moral system. (Often this is phrased in terms of utilitarianism, since the term "utilitarianism" is better known than the more general term "consequentialism", but this is pretty clearly about consequentialism in general.) The argument goes roughly like so: Consequentialism tells us that the thing to do is the thing with the best results. But, this is a ridiculously high standard, that nobody can actually live up to. Thus, consequentialism tells us that everybody is bad, and we should all condemn everybody and all feel guilty. (There are a number of variants.)

This argument is based on a conflation. It assumes that there's one single thing, "morality", and this one thing produces not only answers to "what should you do", but also, what should we condemn, what should we punish, what should one feel guilty about, and other similar questions; and that, moreover, the answers to these questions are identical (or opposites, as appropriate; here the first one would be the opposite of the others).

(Yes, you can add wrinkles like supererogatory acts and such, but that's not particularly relevant to the argument so I'll ignore it.)

But actually, these questions are not identical! The result is that the above reasoning is incorrect. Consequentialism answers one question -- what to do. It is for making decisions. It does not, directly, tell what you should feel guilty about; only what you should do.

But notice that word "directly" there -- since these other questions are also decisions, consequentialism can be used to answer them as well. However, one must answer them as their own decisions, yielding different answers from the conflation above. What should we punish? Whatever it would yield the best result to punish! Not, whichever acts failed to yield the best result and are therefore "bad".

Indeed consequentialism doesn't even have a notion of "good" and "bad" acts, only better ones and worse ones. But t...
Third-wave AI safety needs sociopolitical thinking
by Richard_Ngo — 2025-03-27T00:55:30.548Z

At EA Global Boston last year I gave a talk on how we're in the third wave of EA/AI safety, and how we should approach that. This post contains a (lightly-edited) transcript and slides. Note that in the talk I mention a set of bounties I'll put up for answers to various questions—I still plan to do that, but am taking some more time beforehand to get the questions right.

Transcript and slides

Hey everyone. Thanks for coming. You're probably looking at this slide and thinking: "how to change a changing world? That's a weird title for a talk. Probably it's a kind of weird talk." And the answer is, yes, it is a kind of weird talk.

So what's up with this talk? Firstly, I am trying to do something a little unusual here. It's quite abstract. It's quite disagreeable as a talk. I'm trying to poke at the things that maybe we haven't been thinking enough about as a movement or as a community. I'm trying to do a "steering the ship" type thing for EA.

I don't want to make this too inside baseball or anything. I think it's going to be useful even if people don't have that much EA background. But this seems like the right place to step back and take stock and think: what's happened so far? What's gone well? What's gone wrong? What should we be doing?

So the structure of the talk is:

Three waves of EA and AI safety. (I mix these together because I think they are pretty intertwined, at least intellectually. I know a lot of people here work in other cause areas. Hopefully this will still be insightful for you guys.)
Sociopolitical thinking and AI. What are the really big frames that we need as a community to apply in order to understand the way that the world is going to go and how to actually impact it over the next couple of decades?

There's a thing that sometimes happens with talks like this—maybe status regulation is the word. In order to give a really abstract big picture talk, you need to be super credible because otherwise it's very aspirational. I'm just going to say a bunch of stuff where you might be like, that's crazy. Who's this guy to think about this? It's not really his job or anything. I mean, it kind of is my job, and part of it is my own framework, but a lot of it is reporting back from certain front lines. I get to talk to a bunch of people who are shaping the future or living in the future in a way that most of the world isn't. So part of what I'm doing her...
Knowledge, Reasoning, and Superintelligence
by owencb — 2025-03-26T23:28:11.465Z
Link: https://strangecities.substack.com/p/knowledge-reasoning-and-superintelligence

What makes people good at solving novel problems? In part, it’s having relevant knowledge. And in part, it’s being able to think flexibly to apply that knowledge. So intelligence isn't a scalar: there are at least two dimensions — often called crystallized intelligence and fluid intelligence.

This is also broadly true of AI. We can see the distinction between language models regurgitating facts and reasoning models tracing out implications. We can see the difference between the performance of AlphaGo’s policynet, and what it achieves with Monte Carlo tree search. And we can imagine this distinction even for superintelligent systems.

This is a conceptual discussion. But the motivation is pragmatic. If we have a better understanding of the high-level workings of intelligence, we can make better predictions about what is required for superhuman performance, or what the dynamics of an intelligence explosion might look like. And this could also help us in thinking about how AI systems should be governed — e.g. should we be giving more attention to questions of who controls the creation and distribution of new knowledge?

Crystallized intelligence vs fluid intelligence

For the purposes of this post, intelligence is the ability to process information in ways that are helpful for achieving goals. Intelligent systems typically have both some crystallized intelligence (a body of implicit or explicit knowledge relevant to the task) and some fluid intelligence (being able to think things through on the fly).

In humans, knowing your multiplication tables, knowing how to ride a bike, or playing bullet chess are all chiefly about crystallized intelligence. But proving novel theorems, designing a bicycle from first principles, or playing slower games of chess all depend more on fluid intelligence.

Of course even in these cases crystallized intelligence is an important ingredient for the ability to think things through — e.g. mathematicians are more likely to be able to prove novel theorems if they have a solid grasp of the existing concepts and theorems, rather than needing to rederive them each time. And since our thinking often depends on our lower-level concepts and heuristics, it’s difficult to give clean definitions separating them.

For AI systems, we can be a little more precise. But first we’ll make a couple of related definitions:

Crystallized knowledge refers to data object

...