trentmkelly/LessWrong-43k huggingface dataset:
Superintelligence Alignment Proposal
by Davey Morse — 2025-02-03T18:47:22.287Z

This essay asks whether self-interested superintelligence will ultimately see its self as distinct from or inclusive of humanity, then makes safety recommendations for AI labs.

Instrumental convergence predicts that superintelligence will develop self-interest. Natural selection agrees: the superintelligence which survives the most will be the one that is trying to survive.

Many AI safety experts worry that self interested/survival-oriented superintelligence will look selfish—using humans for its own ends. In Eliezer Yudkowsky’s stark phrase, “The AI does not love you, nor does it hate you, but you are made of atoms it can use for something else.” If an Artificial General Intelligence (AGI) regards us with such indifference, our future looks grim indeed.

But what if self-interest doesn’t imply selfishness? A superintelligent AI’s definition of its “self” could be limited to just its hardware/software/directives—this is the normal assumption. But what if its sense of self included life, in all different forms? What the self it were trying to preserve were not geographically bounded?

This essay first posits two very different ways an AGI might conceive of its own identity—one that leads to competition with humans, and an alternative that could align its interests with ours; then, it introduces reasons that an interconnected self-concept might be not only possible but selected for; and on that basis, it makes simple safety policy recommendations for AGI labs.

The goal of this inquiry, transparently, is driven by a hopeful idea: even a superintelligence driven to survive at all costs, might realize that the best way to preserve itself is to preserve us, too.

Key assumptions:

Superintelligence. This essay assumes a future in which AI has evolved to be 10× more powerful than humans as reasoners, managers, or persuaders. It can procure its own resources, digital and physical, and create ecosystems for survival that don’t depend on reluctant human labor. I’d guess this will be the case in 1–10 years.
Natural Selection Overpowers Human Design. This essay assumes that, in the end, natural selection will shape superintelligence more than any initial design choices by AI developers. In a competitive market where every kind of digital agent will be built and released, the key question is which will last and spread.

The Default: A Discrete, Competitive Self-Concept

It’s...
Gettier Cases [repost]
by Antigone — 2025-02-03T18:12:22.253Z

"The claim that Gettier cases don’t accurately describe the world because they are referentially opaque cuts to the heart of how language smuggles ambiguity into epistemology.

When we say “the person who gets the job has ten coins,” the proposition refers to one thing (e.g., Jones) but mediates that reference through an appeal to the set of all possible things with that quality (e.g., “people with ten coins”).

This creates a sleight of hand: the belief’s justification is anchored to a specific referent (Jones), while its truth depends on a different member of the set (Smith).

The problem isn’t just luck (as others have claimed) [I hope to refute this one day, but I just don't have enough time on my hands at the moment to do so]

Instead -- the constraint to possible referents of an agent’s truth statement is linguistically flexible enough to retroactively validate accidental truths.

Whether these manipulations of language are actually valid or not remains underexplored. I attempt to explore them in my post.

[I tackle this in the original post, but I would like to expand on it further].

If propositions like “the person who…” are inherently semantically unstable—their referents shifting with context—then Gettier cases must be artifacts of linguistic looseness, not genuine paradoxes of truth.

[It is likely someone has come close to this perspective at some point. I think if Wittgenstein were around he would have demolished Gettier cases.]

The question is: are we allowed to “edit” the referent post hoc when evaluating truth, or must propositions rigidly designate their objects?

https://www.lesswrong.com/posts/tLrwLHu2vwMsirzMe/gettier-cases-rigid-designators-and-referential-opacity

Okay, this has received some replies, and some downvotes. This is one of the replies:

"No, there is nothing wrong with the referents in the Gettier examples"

I will have to revisit this assumption when I have done a bit more research into the topic. This is an interesting question that I would like to follow more.

>The problem is not that the proposition refers to Jones.

Who the proposition refers to is always uncertain in Gettier cases -- this is a fundamental fact about Gettier cases. I will safely discard this assumption.

>Within the universe of the scenario, it in fact did not.

This is an error in logic. Who the statement refers to, again, is unclear in the...
Stopping unaligned LLMs is easy!
by Yair Halberstadt — 2025-02-03T15:38:27.083Z

Or at least it might be.

Status: butterfly idea, I have only vague understanding of internals of LLMs

I have no idea how to align superintelligent LLMs. But I think it's quite likely we'll easily be able to catch them in the act and turn them off before they do anything too dastardly.

I'm going to make the following reasonable (but by no means certain) assumptions:

LLMs will never be superintelligent when predicting a single token. Superintelligence emerges when you continuously run an LLM in serial, where it's able to use its earlier output as a scratchpad to solve more complicated problems.
LLMs will have no state. Minus some technical complexities, predicting 10 tokens is no different from predicting one token, using that as input to predict the next token, and so on recursively.
Superintelligent systems based on LLMs may use multiple LLMs in all sorts of complex configurations, but not in a way that breaks the above assumptions about LLMs, and not in a way that any emergent behaviour of the system as a whole isn't reflected in the outputs of any of the constituent LLMs.

If so, it seems that all you need to do to detect any unwanted behaviour from a superintelligent system is to feed all output from constituent LLMs to a simpler LLM to detect output that looks like it's leading towards unaligned behaviour. Only once the output has been verified, pass it on to the next system (including looping it back to itself to output more tokens). If it fails verification immediately stop the whole system.

Objections

But won't the superintelligent AI be able to trick the verifier?

The superintelligent AI definitely could trick the verifier. But the verifier isn't checking the superintelligent AI, it's checking each individual pass of the LLM. Since each individual pass is not superintelligent, if the AI decided to trick the verifier it would have decide to do so statefully - in the LLM output. So the verifier can see the AIs decision before that decision has been passed anywhere, and can shut down the superintelligent AI.

Effectively the verifier acts before the system is yet superintelligent.

But won't the superintelligent AI prevent itself being shut down?

Same answer. It definitely could, but it would have to decide to do so, and by then the LLM will already have shut it down.

Akkkshully LLMs generate multiple tokens per pass, and what if they use multi headed quantronmium flip ...
The Outer Levels
by Jerdle — 2025-02-03T14:30:29.230Z

Required knowledge: Simulacrum Levels.

There is a distinct degree of confusion around simulacrum level 4, where the original definition (or related ones) seems to have been replaced with a new concept that is related to 3 in the same way 2 is related to 1. In this new form, the true nature of simulacrum level 4 is obscured, because it would actually belong to a higher level. These potential levels will be discussed, as will an even lower level than the usual lowest.

Existing levels

Level 1

This is probably the simplest to explain. Level 1 is about truth and factual accuracy, trying to create an accurate map of reality. But it still creates a map, rather than being the territory.

Level 2

Level 2 is not a level with significant controversy as to the meaning. It is the level of lies. Or, more generally, it is the level in which statements are treated as tracking truth (like on level 1), but level 2 actors use this to act on people's maps of the world and factual beliefs. Some of the level 2 "lies" can be true (as can all higher levels), but they are not told because they are true, but because they will be believed.

As such, level 2 is parasitic on the level 1 structure. As with other parasites, it still relies on that structure's general acceptance, and so is in the same tier as level 1. This I will refer to as the factual tier, as statements are treated as reflecting factual beliefs.

With the lion analogy, yes, the level 2 meaning of "There's a lion across the river" is "I don't want to go across the river", but it is important that it is interpreted as "There is an actual lion across the actual river that will actually eat you".

Level 3

Level 3 is also generally accepted as the tribal/social level, where the facts no longer matter, but it is important to send accurate signals of your views. Whether the statement is true or false is irrelevant, and the literal meaning is no longer parsed, but the symbolic meaning is expected to be genuine. The tier this works on is the political tier, that of slogans and support for a side.

Original

The original meaning of level 3 is rather different. Going to the pseudo-source text (this is actually a Wikipedia description, rather than Baudrillard himself - there's a certain irony in that our concept of simulacra is so much a simulacrum of the original):

The third stage masks the absence of a profound reality, where the sign pretends to b

...
o3-mini Early Days
by Zvi — 2025-02-03T14:20:06.443Z

New model, new hype cycle, who dis?

On a Friday afternoon, OpenAI was proud to announce the new model o3-mini and also o3-mini-high which is somewhat less mini, or for some other reasoning tasks you might still want o1 if you want a broader knowledge base, or if you’re a pro user o1-pro, while we want for o3-not-mini and o3-pro, except o3 can use web search and o1 can’t so it has the better knowledge in that sense, then on a Sunday night they launched Deep Research which is different from Google’s Deep Research but you only have a few of those queries so make them count, or maybe you want to use operator?

Get it? Got it? Good.

Yes, Pliny jailbroke o3-mini on the spot, as he always does.

This most mostly skips over OpenAI’s Deep Research (o3-DR? OAI-DR?). I need more time for that. I’ll cover o3-DR properly later in the week once we have a chance to learn what we’ve got there, along with the non-DR ‘one more thing’ Altman is promising. So far it looks super exciting, but it’s a very different class of product.

Table of Contents

Feature Presentation.

Q&A.

The Wrong Side of History.

The System Card.

The Official Benchmarks.

The Unofficial Benchmarks.

Others Report In.

Some People Need Practical Advice.

Feature Presentation

What exactly can o3-mini do?

OpenAI: We’re releasing OpenAI o3-mini, the newest, most cost-efficient model in our reasoning series, available in both ChatGPT and the API today. Previewed in December 2024⁠, this powerful and fast model advances the boundaries of what small models can achieve, delivering exceptional STEM capabilities—with particular strength in science, math, and coding—all while maintaining the low cost and reduced latency of OpenAI o1-mini.

OpenAI o3-mini is our first small reasoning model that supports highly requested developer features including function calling⁠(opens in a new window), Structured Outputs⁠(opens in a new window), and developer messages⁠(opens in a new window), making it production-ready out of the gate. Like OpenAI o1-mini and OpenAI o1-preview, o3-mini will support streaming⁠(opens in a new window).

Also, developers can choose between three reasoning effort⁠(opens in a new window) options—low, medium, and high—to optimize for their specific use cases.

They’re all in the API. Who gets chatbot access? To some extent, everyone.

ChatGPT Plus, Team, and Pro users can access OpenAI o3-mini

...
OpenAI releases deep research agent
by Seth Herd — 2025-02-03T12:48:44.925Z
Link: https://openai.com/index/introducing-deep-research/

Experimental research project record here

Edit: the most important question raised in the comments was: How much of this capability actually comes from the end-to-end task-based RL on CoT, and how much just from a better prompting scheme of "ask the user for clarifications, do some research, then think about that research and decide what further research to do?"

This matters because end-to-end RL seems to carry greater risks of baking in instrumental goals. It appears that people have been trying to do such comparisons: Hugging Face researchers aim to build an ‘open’ version of OpenAI’s deep research tool, and the early answer seems to be that even o1 isn't quite as good as the Deep Research tool - but that could be because o3 is smarter, or the end-to-end RL on these specific tasks.

Back to the original post:

Today we’re launching deep research in ChatGPT, a new agentic capability that conducts multi-step research on the internet for complex tasks. It accomplishes in tens of minutes what would take a human many hours.

Deep research is OpenAI's next agent that can do work for you independently—you give it a prompt, and ChatGPT will find, analyze, and synthesize hundreds of online sources to create a comprehensive report at the level of a research analyst. Powered by a version of the upcoming OpenAI o3 model that’s optimized for web browsing and data analysis, it leverages reasoning to search, interpret, and analyze massive amounts of text, images, and PDFs on the internet, pivoting as needed in reaction to information it encounters.

The ability to synthesize knowledge is a prerequisite for creating new knowledge. For this reason, deep research marks a significant step toward our broader goal of developing AGI, which we have long envisioned as capable of producing novel scientific research.

...

How it works

Deep research was trained using end-to-end reinforcement learning on hard browsing and reasoning tasks across a range of domains. Through that training, it learned to plan and execute a multi-step trajectory to find the data it needs, backtracking and reacting to real-time information where necessary. ... As a result of this training, it reaches new highs on a number of public evaluations focused on real-world problems.

Note that the pass rates are around 15-25%, so it's not outright replacing any experts—and it remains to be seen whether the results are even ...
Neuron Activations to CLIP Embeddings: Geometry of Linear Combinations in Latent Space
by Roman Malov — 2025-02-03T10:30:48.866Z

This is my project for the AI Safety Fundamentals course.

Abstract

I use Lucent library to produce images that are optimized to give maximum activation on some neuron (or, in my case, linear combination of two neurons). Then I use CLIP neural network to map those images in the embedding space. Then, I use dimensionality reduction and look for interesting geometry. Here is the code I used.

Getting images

Lucent already has a built-in way to combine objectives (in this case, activations of 2 neurons). But which objectives should it optimize for? At first, I thought to take a grid of linear combinations

x and y values for obj=x⋅neuron1+y⋅neuron2

But this is not natural way to do that. It doesn't matter whether obj1=x⋅neuron1+y⋅neuron2 is optimized or obj2=2obj1=2x⋅neuron1+2y⋅neuron2, the result is going to be the same (because f(x) and 2f(x) have the same point of maximum). So I only need to look at linearly independent linear combinations, or directions. I am using circular parametrization x=sin(ϕ),y=cos(ϕ)

Played back to back, those optimized images make a GIF (the result is flashy so epilepsy warning, the GIF is in the end of the post). Four distinct phases can be noticed, which I assume are related to which neuron has the biggest (in absolute value) coefficient and whether it is being maximized or minimized.

CLIP embeddings

CLIP is a model that is used primarily in order to have a shared embedding space for images and image descriptions. I use it just as a model which gives me low-dimensional vector corresponding to the input image.

But in order to visualize them, those vectors need dimensionality reduction.

Two methods I used:

PCE

Using PCE to reduce 768 dimensions to 2, we get this image

It is circular, as to be expected, though it has 3 distinct clusters (top right, top left, and at the bottom)

Get top 2 dimensions with biggest variance.

This method results in following image

Now it is tangled, but that is probably just an artifact of this particular projection (using the right hyperplane, as in example above, no tangles are formed). Just as in previous examples, 3 distinct clusters can be noticed.

Next steps

Here are a few possible and easy additions to the project

Experiment with more neurons / visualize more dimensions of the CLIP vectors.
Try different dimensionality reduction techniques.
Explore what

...
Can we infer the search space of a local optimiser?
by Lucius Bushnaq — 2025-02-03T10:17:01.661Z

Suppose we have a system that we suspect is some form of local optimiser. Maybe it's gradient descent training a neural network. Maybe it’s a neural network doing in-context learning. Maybe it's a mind, because we guess that the operation of minds in general is to an extent well-described as local movement in some 'abstraction space' a lot of the time. Maybe it’s something else.

Is there, on paper, a general way:

To check whether our guess that the system is a local optimiser is correct?
To find the space in which the optimiser is taking local steps?

Setup

Concretely, say a system has a state vector s, which we can see. s evolves over time steps t, so let’s call it st. Our guess is that the system evolves st→st+1 through some local optimisation method. But it’s not really local in s-space. It’s local in some other space we don’t know about. Let’s call the state vector in that hidden space we don’t know about ht∈H, where H is some space. H seems guaranteed to be a topological space, else the idea of a local search in it wouldn't make much sense. Maybe in a lot of cases we can also assume that it is a metric space? Not sure about that one, but a lot of the local search algorithms I can think of seem to have some notion of distance between points in the space.

So, overall the system maps st→st+1 as st→ht→ht+1→st+1, where the transformations st→ht and ht+1→st+1 can be complicated and nonlinear.

ht→ht+1 is assumed to be a step in some local optimisation method. That is to say, there is some evaluation function f(ht) that the system is trying to minimise, and it tries to do this minimisation via something like a genetic algorithm, gradient descent, Adam, or something else that makes use of local information to take one step in the space H per time step t. The evaluation function f(h) should have the kind of loss landscape a local optimiser could reasonably navigate. No parity learning problems or anything like that. It also shouldn’t be completely trivial to find the minimum of f(h). I.e. if the optimiser is using Newton’s method, f(h) shouldn’t just be quadratic everywhere, else there isn’t much of a search trajectory for us to look at because the system always finds the minimum in one step.

Now, what I’m wonderi...
Pick two: concise, comprehensive, or clear rules
by Screwtape — 2025-02-03T06:39:05.815Z

There once was a post on LessWrong asking what's with all the bans? The OP of that post complained of, among other things, being punished despite not having broken any rules that they knew about before taking the action they got punished for. I started to write a comment, which then ballooned out of scope. The following is based on my own experiences trying to write rules.

I.

I claim the ideal rules are concise, comprehensive, and clear[1]. In practice, I think that we usually have to pick two.

Concise: Ideal rules are short. You can read or refer to them easily.

Comprehensive: Ideal rules include everything relevant. There aren't gaps.

Clear: Ideal rules have one interpretation. There's no ambiguity.

If you're running LessWrong, then "Don't be a jerk, talk about things related to rationality" is a concise and fairly comprehensive set of rules. It's just not a very clear set of rules. There will be myriad places where people disagree with what counts as being a jerk or what relates to rationality. "Do unto others as you would have them do unto you" is a famous concise and comprehensive rule with a lot of interpretations. If you aren't under a lot of heated or adversarial pressure though, concise and comprehensive rules are great! When they fail, it's because people don't agree on how to interpret the rules.

There's a subtype of concise and comprehensive that goes like this: "The moderators shall do whatever they want." This is very unclear! It is impressively comprehensive however, since your mods can do whatever they think is reasonable whenever an issue is brought to their attention. (Even easier if they don't need to be bound by precedent.) It doesn't scale especially well, since it's hard to automate mod judgement. Eventually your mods might disagree with each other, a problem which can be solved by the second rule "the moderators shall do whatever they want, unless overruled by the administrator."

The official rules for Magic: The Gathering are comprehensive and clear. Magic is played competitively by an often pedantic fanbase, the more competitive games are overseen by trained judges, and its rules are programmed into a computer which accepts no on-the-fly rulings. The comprehensive rules are also (at the time of this writing) two hundred and ninety three pages long and include blocks of text like the following:

109.5. The words “you” and “your” on an obj

...
Language Models and World Models, a Philosophy
by kyjohnso — 2025-02-03T02:55:36.577Z
Link: https://hylaeansea.org/blog/2025/02/01/World-Models-and-Language-Models.html

This is my submission for Blue Dot Impact's AI Alignment Fundamentals Course. In this post I explore computational policy, collaborative development of policy with AI, and how linking Language Models with World Models can lead to more precise training data, novel insights into the policy domain, and potentially new knowledge.