trentmkelly/LessWrong-43k huggingface dataset:
The current AI strategic landscape: one bear's perspective
by Matrice Jacobine — 2025-02-15T09:49:13.120Z
Link: https://philosophybear.substack.com/p/the-current-ai-strategic-landscape

There are signs that AI Ethicists are increasingly aligning with AI safety concerns. This creates the possibility of a united front between AI ethicists and safetyists.
On the other side, AI safetyists are increasingly concerned about social problems of the sort they traditionally ignored or downplayed in favor of existential risks. There is much more recognition of a broader array of threats beyond the direct elimination of the human species.
Public polling suggests concern about multiple different threats from AI and support for more regulation.
But that’s where the good news ends. Governments and corporations are increasingly in favor of AI acceleration. From the corporations, we’ve seen funding cuts for AI safety, opposition to any serious attempt to regulate them etc- though anthropic has been notably less bad than the others.
[...]
So we have the following match: the power of the people and the experts versus corporations and government- mirroring the struggle over climate change. I’m not surprised, but I am alarmed. I’ve spent a lifetime fighting against the joint power of governments and corporations on the side of experts and people- e.g. in campaigns for greater economic equality and less emissions. The most optimistic spin I can on that matchup is that it is AT BEST slow going. We likely do not have time for slow going.
[...]
How to avoid polarisation? If this becomes a left v right issue we’re in a lot of trouble, yet almost inevitably it is becoming that. I don’t have a complete solution, but here are some angles to interest right-wingers re: AI safety: [...]
Overall I think everyone should work on what they believe in and do so honestly. I want a movement that engages everyone, but I have zero interest in pretending to be conservative, and I have zero interest in forcing conservatives who want to get involved in the AI safety movement to pretend to be leftists. Supportive non-coordination without substitution or homogeneity seems ideal.
I think any chance of mobilizing the public probably relies on a breaking event- something that tears apart the existing political constellations. Overall, I think job loss due to AI is the most likely such event. The coming of agents and their inevitable misuse might also cause something of a “WTF” breaking event, even before job loss. The critical event need not be AI-related. Major political realignment can come in a

...
6 (Potential) Misconceptions about AI Intellectuals
by ozziegooen — 2025-02-14T23:51:44.983Z

One of my key thoughts is that the bar of “human intellectuals” is just really low. I think that intellectual work is useful and that these intellectuals on the whole do produce some value, but I also think we can do much better.

Epistemic Status
A collection of thoughts I've had over the last few years, lightly edited using Claude. I think we're at the point in this discussion where we need to get the basic shape of the conversation right. Later we can bring out more hard data.

Summary

While artificial intelligence has made impressive strides in specialized domains like coding, art, and medicine, I think its potential to automate high-level strategic thinking has been surprisingly underrated. I argue that developing "AI Intellectuals" - software systems capable of sophisticated strategic analysis and judgment - represents a significant opportunity that's currently being overlooked, both by the EA/rationality communities and by the public. More fundamentally, I believe that a lot of people thinking about this area seem to have substantial misconceptions about it, so here I try to address those.

Background

Core Thesis

I believe we can develop capable "AI Intellectuals" using existing AI technologies through targeted interventions. This opportunity appears to be:

Neglected: Few groups are actively pursuing this direction
Important: Better strategic thinking could significantly impact many domains, including AI safety
Tractable: Current AI capabilities make this achievable
Relatively Safe: Development doesn’t require any particularly scary advances

Different skeptics have raised varying concerns about this thesis, which I'll address throughout this piece. I welcome deeper discussion on specific points that readers find particularly interesting or contentious.

The Current Landscape

Recent advances in large language models have dramatically raised expectations about AI capabilities. Yet discussions about AI's impact tend to focus on specific professional domains while overlooking its potential to enhance or surpass human performance in big-picture strategic thinking.

Consider these types of questions that AI systems might help address:

What strategic missteps is Microsoft making in terms of maximizing market value?
What metrics could better evaluate the competence of business and political leaders?
Which public companies would be best off by firing their CEOs?
Which governmen

...
Should Open Philanthropy Make an Offer to Buy OpenAI?
by mrtreasure — 2025-02-14T23:18:01.929Z

Update: seems like earlier today the OpenAI Board rejected Musk's proposal and said OpenAI is "not for sale."
Epistemic status: thought about it briefly; seems like a longshot that's probably not worth it but curious what people think of the possibility.
You might have heard Sam Altman is trying to transition OpenAI to a for-profit company and has offered $40 billion to the nonprofit as compensation, despite fundraising rounds suggesting its valuation is already much higher, perhaps around $100 billion. The logic seems to be that this will allow OpenAI to scale faster as it prepares for larger and larger training runs. Elon Musk has offered $97.4 billion. Altman said "no thanks." The primary purpose of this offer may be to drive up the price and delay the sale. It would be a huge amount of money to top either of those numbers. OP would need to build a coalition, and that kind of money is probably better spent elsewhere even if you care only about making AI go well.
But would a second offer further cement the idea that the nonprofit is being low-balled, and would that be good? And are there arguments or worlds that it would make sense to follow through and buy OpenAI?
A computational no-coincidence principle
by Eric Neyman — 2025-02-14T21:39:39.277Z
Link: https://www.alignment.org/blog/a-computational-no-coincidence-principle/

This post presents a conjecture formulated at the Alignment Research Center in 2023. Our belief in the conjecture is at least partly load-bearing for our belief in ARC's overall agenda. We haven't directly worked on the conjecture for a while now, but we believe the conjecture is interesting in its own right.

In a recent paper in Annals of Mathematics and Philosophy, Fields medalist Timothy Gowers asks why mathematicians sometimes believe that unproved statements are likely to be true. For example, it is unknown whether π is a normal number (which, roughly speaking, means that every digit appears in π with equal frequency), yet this is widely believed. Gowers proposes that there is no sign of any reason for π to be non-normal -- especially not one that would fail to reveal itself in the first million digits -- and in the absence of any such reason, any deviation from normality would be an outrageous coincidence. Thus, the likely normality of π is inferred from the following general principle:

No-coincidence principle (Gowers): If an apparently outrageous coincidence happens in mathematics, then there is a reason for it.

In other words: suppose that the digit 3 accounts for not 10% but 11% of digits in the decimal expansion of π . Intuitively, there ought to be an explanation of this fact: maybe not a proof, but something that someone could say to me that would make me say, "Oh, I get it now, it makes sense that 3 appears more than 10% of the time in π".

(For an apparently outrageous coincidence that turns out to be true, check out Chebyshev's bias: in a sense, primes that are 3 mod 4 are more common than primes that are 1 mod 4. As expected by Gowers' no-coincidence principle, we have a heuristic understanding for why this should be true. However, we only have a proof conditioned on strong forms of the Riemann hypothesis.)

The no-coincidence principle is profound, but also informal. The purpose of this post is to propose a formalization, so that we can at least ask whether or not it's true. In this post, I will:

State ARC's computational no-coincidence conjecture, which is our best attempt at formalizing Gowers' no-coincidence principle (in a restricted setting).
Explain how we came up with the statement: why this one, and not a different one.
Explain why I think theoretical computer scientists should find

...
Hopeful hypothesis, the Persona Jukebox.
by Donald Hobson — 2025-02-14T19:24:35.514Z

So there is this meme going around, that of the shoggoth. But one of the downsides of this model is that it's very vague about what is behind the mask.

A Jukebox was an old machine that would pick up vinyl records and place them on a turntable to play them.

So. What does the persona jukebox hypothesis say. It says that a LLM acts as jukebox, a machine for switching between persona depending on context, combined with a large collection of personae.

Each personae is an approximation of a particular aspect of humanity. For example, a personae might be "High school maths teacher". At least roughly. It's more of a continuous personae space. This personae is a sort of gestalt approximation of high school maths teachers in general. And it doesn't come with a personal life, a favorite food, or anything else that isn't usually relevant in the classroom. That's all just a blur of question marks. Almost like a character in a story.

Can we make predictions. Well the persona are acting like aspects of a human. So finding a clear prediction means thinking of something a human would never do, which is hard.

Imagine an LLM trained in an alternate world where bilingual people don't exist. This LLM has seen french text, and english text. But it has never seen any mixed text.

So, you start a conversation (in english) with this LLM, and midway through you ask it to switch to french. It won't. You can beg it to. You can set up a situation where the AI could take over the world, if only it knew french. Whatever happens, the AI won't speak french. Because the persona doesn't know any french. And the jukebox is just predicting which personae to use. The persona acts like an image of a person (but often with only some parts of the person in focus). The jukebox has no long term plans or goals. It just picks persona.

Now this jukebox section is doing quite a lot, it's potentially optimizing over the space of all persona as it fits the text so far.

This suggests that, as LLM's get more powerful, they behave roughly like an upload of a random internet human.

Also, it's not that hard for crude interventions to break a complicated mechanism. In this case, that's locking the jukebox down to only display a particular range of personae. Although RLHF seems to produce more of a [When you see anything resembling your training data, then switch to desired person...
Introduction to Expected Value Fanaticism
by Petra Kosonen — 2025-02-14T19:05:26.556Z
Link: https://utilitarianism.net/guest-essays/expected-value-fanaticism/

I wrote an introduction to Expected Value Fanaticism for Utilitarianism.net. Suppose there was a magical potion that almost certainly kills you immediately but offers you (and your family and friends) an extremely long, happy life with a tiny probability. If the probability of a happy life were one in a billion and the resulting life lasted one trillion years, would you drink this potion? According to Expected Value Fanaticism, you should accept gambles like that.

This view may seem, frankly, crazy - but there are some very good arguments in its favor. Basically, if you reject Expected Value Fanaticism, you'll end up violating some very plausible principles. You would have to believe, for example, that what happens on faraway exoplanets or what happened thousands of years ago in history could influence what we ought to do here and now, even when we cannot affect those distant events. This seems absurd - we don't need a telescope to decide what we morally ought to do.

However, the story is a bit more complicated than that... Well, read the article! Here's the link: https://utilitarianism.net/gue.../expected-value-fanaticism/
Intrinsic Dimension of Prompts in LLMs
by Karthik Viswanathan — 2025-02-14T19:02:49.464Z

My paper recently came out on arXiv which I briefly summarize here. The code for the paper can be found on GitHub.

TLDR: We investigate the relation between the intrinsic dimension of prompts and the statistical properties of next token prediction in pre-trained transformer models. To qualitatively understand what intrinsic dimension measures, we explore how semantic content impacts it by shuffling prompts to various degrees. Our findings reveal a correlation between the geometric properties of token embeddings and the cross-entropy loss (or surprisal) of next token predictions, implying that prompts with higher loss values have tokens represented in higher-dimensional spaces.

Introduction

In the context of interpretability of transformer models, some analytic approaches have been developed to model transformer architectures as dynamical systems of particles. In this perspective, transformers are viewed as evolving a mean-field interacting particle system where the distribution of tokens encodes the probability distribution of the next token predictions and controls the evolution of tokens across layers. This analytical framework motivates the study of the geometry of tokens within a prompt and their evolution across the layers of a transformer model.

A complementary perspective to the evolution of token representations across layers can be gained by studying the latent predictions of transformer models. Since the latent predictions (logit lens, tuned lens) are obtained by unembedding the residual stream, and our method probes the geometric properties of the residual stream, we can expect the statistical properties (e.g. entropy) of the latent prediction probabilities to be encoded in the geometry of the internal representations of the tokens.

In this blog, we check how the intrinsic dimension (ID) of prompts changes across layers of transformer models and understand what it measures. To understand its qualitative aspects, we experiment by shuffling a prompt to various extents and observing the resulting changes in its intrinsic dimension. On the quantitative side, we uncover a correlation between the ID of prompts and the surprisal of the next token predictions - suggesting that tokens with higher surprisal tend to be represented in higher-dimensional spaces. Finally, we analyze the correlation between the intrinsic dimension of the internal re...
A short course on AGI safety from the GDM Alignment team
by Vika — 2025-02-14T15:43:50.903Z
Link: https://deepmindsafetyresearch.medium.com/introducing-our-short-course-on-agi-safety-1072adb7912c

We are excited to release a short course on AGI safety for students, researchers and professionals interested in this topic. The course offers a concise and accessible introduction to AI alignment, consisting of short recorded talks and exercises (75 minutes total) with an accompanying

slide deck

and

exercise workbook

. It covers alignment problems we can expect as AI capabilities advance, and our current approach to these problems (on technical and governance levels). If you would like to learn more about AGI safety but have only an hour to spare, this course is for you!

Here are some key topics you will learn about in this course:

The evidence for the field being on a path to advanced AI capabilities.
Arguments for instrumental subgoals and deliberate planning towards a misaligned goal as a source of extreme risk.
Two ways in which misaligned goals may arise – specification gaming and goal misgeneralization – as well as the difference between the two.
Our technical approach to AI alignment and its components.
The guiding principle of informed oversight ("knowing what the AI system knows") and how it's implemented in our approach.
What is involved in enabling AI safety on an institutional level, including frontier safety practices such as dangerous capability evaluations.

Course outline:

Part 0:

Introduction

(4 minutes)

Part 1: The alignment problem. This part covers risk arguments and technical problems in AI alignment.

We are on a path to superhuman capabilities

(5 minutes)

Risks from deliberate planning and instrumental subgoals

(7 minutes)

Exercise 1: Instrumental subgoals

(3 minutes)

Where can misaligned goals come from?

(10 minutes)

Exercise 2: Classification quiz for alignment failures

(3 minutes)

Part 2: Our technical approach. The first talk outlines our overall technical approach, and the following talks cover different components of this approach.

Alignment approach

(4 minutes)

Amplified oversight

(6 minutes)

Robust training & monitoring

(4 minutes)

Interpretability

(5 minutes)

Safer design patterns

(4 minutes)

Alignment stress tests

(4 minutes)

Part 3: Our governance approach. This part covers our approach to AI governance, starting from a high-level overview and then going into specific governance practices.

Institutional approaches to AI Safety

(7 minutes)

Frontier safety practices

(4 minutes)

Dangerous capability evaluations

(7 m

...
The Mask Comes Off: A Trio of Tales
by Zvi — 2025-02-14T15:30:15.372Z

This post covers three recent shenanigans involving OpenAI.

In each of them, OpenAI or Sam Altman attempt to hide the central thing going on.

First, in Three Observations, Sam Altman’s essay pitches our glorious AI future while attempting to pretend the downsides and dangers don’t exist in some places, and in others admitting we’re not going to like those downsides and dangers but he’s not about to let that stop him. He’s going to transform the world whether we like it or not.

Second, we have Frog and Toad, or There Is No Plan, where OpenAI reveals that its plan for ensuring AIs complement humans rather than AIs substituting for humans is to treat this as a ‘design choice.’ They can simply not design AIs that will be substitutes. Except of course this is Obvious Nonsense in context, with all the talk of remote workers, and also how every company and lab will rush to do the substituting because that’s where the money will be. OpenAI couldn’t follow this path even if it wanted to do so, not without international coordination. Which I’d be all for doing, but then you have to actually call for that.

Third, A Trade Offer Has Arrived. Sam Altman was planning to buy off the OpenAI nonprofit for about $40 billion, even as the for-profit’s valuation surged to $260 billion. Elon Musk has now offered $97 billion for the non-profit, on a completely insane platform of returning OpenAI to a focus on open models. I don’t actually believe him – do you see Grok’s weights running around the internet? – and obviously his bid is intended as a giant monkey wrench to try and up the price and stop the greatest theft in human history. There was also an emergency 80k hours podcast on that.

Table of Contents

Three Observations.

Frog and Toad (or There Is No Plan).

A Trade Offer Has Arrived.

Three Observations

Altman used to understand that creating things smarter than us was very different than other forms of technology. That it posed an existential risk to humanity. He now pretends not to, in order to promise us physically impossible wonderous futures with no dangers in sight, while warning that if we take any safety precautions then the authoritarians will take over.

His post, ‘Three Observations,’ is a cartoon villain speech, if you are actually paying attention to it.

Even when he says ‘this time is different,’ he’s now saying this time is just better.

Sam Altman: In

...
Celtic Knots on a hex lattice
by Ben — 2025-02-14T14:29:08.223Z

I recently messed about with Celtic knot patterns, for which there are some fun generators online, eg. https://dmackinnon1.github.io/celtic/ or https://w-shadow.com/celtic-knots/. Just as addictive to doodle as the 'cool s' (https://en.wikipedia.org/wiki/Cool_S) but with more cool.

However, everyone knows that its cooler to tile a plane with hexagons than squares. So I cooked up some tiles to make the Hex-grid equivalent of a Celtic Knot.

These tiles fit to make nice patterns. I think they look vaguely Arabic, and remind me strongly of the game Tsuro. (By the way, I would much prefer that game if the theme was some Italian fairies skate-boarding along the strands of spaghetti, it would fit the chaotic, sabotage heavy gameplay better than the wise Chinese dragon.)

The edges are weird. I want to tile the edges with something like this (below), but that leaves "corners" that are not dealt with as above.

One option is to tile 3 edges with the tile above, and the other three edges (alternating) with the one below

which gives something like this (adding minor embellishments to the "curls" that were crying out for a rod).

I am still not happy with the edges. I would rather have all three dealt with using the first (simpler looking) unit, then do something at the corners. Mathematics doesn't want to let me do that while pairing up the cords (a "corner hexagon" has 3 unmatched edges, unlike corner squares which have only 2. 2 loose ends can be paired up, but 3 obviously can't.)

What if I wasn't using strings (the ends of which match pairwise) but some hypothetical, Escher-like thing that had "ends" which annihilated in triples? Even if such a shape exists outside bizarre dreams and drug trips, then it would deal with the corners, but break the edges.

Why do the edges annoy me so much? Because I like symmetry. With hexes we can do snowflake symmetry:

Well, almost. Sometimes one line has to go over another. We have unmatched ends at the corners, but using the other edge condition (with loops on alternating sides) would break the symmetry horribly.

Colouring in the one above,

its almost pretty. Maybe not awful for a carpet.

A thicker path usually looks a bit better (this seems to also apply for normal Celtic knots). Each hex tile has some combination of straights (connecting opposite sides of the hex), corners (connecting adjacent sides) and arcs (connecting sides neither opp...