trentmkelly/LessWrong-43k huggingface dataset:
The need to relativise in debate 
by Geoffrey Irving — 2025-06-26T16:23:48.995Z

Summary: This post highlights the need for results in AI safety, such as debate or scalable oversight, to 'relativise', i.e. for the result to hold even when all parties are given access to a black box 'oracle' (the oracle might be a powerful problem solver, a random function, or a model of arbitrary human preferences). Requiring this relativisation can often significantly change the complexity class associated with a given protocol and therefore the promise of a given method. We present a few examples of this change and the solutions required to restore the methods back to full power.

Goal: self-contained intuition

This is a purely expositional post describing some of the intuition for why we need relativisation to apply interactive proof techniques to AI safety, and what that does to the necessary proof systems. No new research results, but I previously hadn’t written this story down in a clean way, so now I have something to point to.

Interactive proof systems and IP = PSPACE

An interactive proof system consists of a verifier V and one or more provers P, such that the provers are trying to convince the verifier of answers to questions (formally, usually membership of strings x in some language L). The simplest proof system is the non-interactive one: a single prover produces a proof string y=P(x), and the verifier checks the proof as V(x,y). This corresponds to the complexity class NP.

There are more complicated proof systems, reaching much higher complexity classes. With polynomially many rounds of interaction, a single prover can reach the complexity class PSPACE, which is the celebrated result

IP = PSPACE

(IP = interactive proofs). However, this result requires re-coding the input problem as a statement about polynomials over finite fields, then using randomness to do polynomial identity testing in the finite field case. For example, if our PSPACE problem is the alternating Boolean circuit ∃x0∀y0…∃xm∀ynϕ(x,y) where ϕ(x,y) is a circuit, we form an interactive proof by replacing x∧y→xy,x∨y→1−(1−x)(1−y), etc. (and doing some trickery to lower degrees, which I will ignore for this sketch). The prover then makes claims about what the summary polynomials are if we marginalise over some suffix of the variables. If the finite field is large enough, a single randomly chose...
Proverbial Corollaries
by Jordan Rubin — 2025-06-26T15:25:01.253Z
Link: https://jordanmrubin.substack.com/p/proverbial-corollaries

Proverbs are pocket‑sized IF‑statements: folk pseudocode that ships the trigger but hides the ELSE.

To find alpha: treat the proverb as true, then ask what happens in the missing branch.

1 RECIPROCITY

The sayings in this bucket assume social payoffs are maximized when inputs and outputs match.

“Treat others as you wish to be treated.”

Implication: The way they treat you is the way they want to be treated by you.
Mirror their style. Tell stories to the storyteller. Ask the question-asker questions. Criticize the critic.
Warm feedback? Great. Cold recoil? You are in a non-reciprocal setting.

“Give credit where credit is due.”

Implication: Undue credit is a loan. Expect to repay it.
When you receive unearned praise, log it as an attempted leverage hook. Bank the flattery, audit the motive, and don’t accept the IOU until verified.

“Honesty is the best policy.”

Implication: Dishonesty harms somewhere. Figure out where.
Map the system by modeling precisely how dishonesty would destroy you. Failure points reveal where the policy actually matters.

“Absence makes the heart grow fonder.”

Implication: If your heart’s not fonder, they weren’t absent enough.
Too-near absence is just muted presence. Extend the gap until longing spikes, or no-one cares anymore. Clarity either way.

Anyone who enacts one of these norms is broadcasting a personal desire. Echo to confirm; if they recoil, you’ve unmasked a performance.

2 THRESHOLDS

These proverbs embed a hidden boundary, x ≥ k, beyond which the advice reverses.

“A bird in the hand is worth two in the bush.”

Implication: A bird in the hand is worth less than three in the bush.
The risk‑adjusted math flips once the opportunity is big enough. Trade away the sure thing.

“A picture is worth a thousand words.”

Implication: A picture is worth less than ten thousand words.
When a 10,000‑word technical doc costs pennies to create, store, and search, prefer text. Pictures smuggle ambiguity.

“Look before you leap.”

Implication: If others are leaping, assume they looked.
A mass of people already leaping marks discovered value. Verify quickly, then jump while uncertainty is still cheap.

“No pain, no gain.”

Implication: If there’s no gain, there should be no pain.
Discomfort untracked by progress is malpractice. Make sure your pain is paying you.

There is a point beyond which proverbial guidance still applies, but in reverse. Locate that point; once crossed, do the opposit...
The Industrial Explosion
by rosehadshar — 2025-06-26T14:41:41.238Z
Link: https://www.forethought.org/research/the-industrial-explosion

Summary

To quickly transform the world, it's not enough for AI to become super smart (the "intelligence explosion").

AI will also have to turbocharge the physical world (the "industrial explosion"). Think robot factories building more and better robot factories, which build more and better robot factories, and so on.

The dynamics of the industrial explosion has gotten remarkably little attention.

This post lays out how the industrial explosion could play out, and how quickly it might happen.

We think the industrial explosion will unfold in three stages:

AI-directed human labour, where AI-directed human labourers drive productivity gains in physical capabilities.

We argue this could increase physical output by 10X within a few years.

Fully autonomous robot factories, where AI-directed robots (and other physical actuators) replace human physical labour.

We argue that, with current physical technology and full automation of cognitive labour, this physical infrastructure could self-replicate about once per year.
1-year robot doubling times is very fast!

Nanotechnology, where physical actuators on a very small scale build arbitrary structures within physical limits.

We argue, based on experience curves and biological analogies, that we could eventually get nanobots that replicate in a few days or weeks. Again, this is very fast!

Intro

The incentives to push towards an industrial explosion will be huge. Cheap abundant physical labour would make it possible to alleviate hunger and disease. It would allow all humans to live in the material comfort that only the very wealthiest can currently achieve. And it would enable powerful new technologies, including military technologies, which rival states will compete to develop.

The speed of the industrial explosion matters for a few reasons:

Some types of technological progress might not accelerate until after the industrial explosion have begun, because they are bottlenecked by physical infrastructure for running experiments
If the industrial explosion is fast, a nation that leads on AI and robotics could quickly develop a decisive economic and military advantage over the rest of the world.
All things equal, a faster industrial explosion means the AI could overthrow humanity sooner – many routes to AI takeover go via AI controlling new physical infrastructure.
The industrial explosion will feed back into the intelligence explosi

...
The Iceberg Theory of Meaning
by Richard Juggins — 2025-06-26T12:13:31.584Z

[Crossposted from my substack Working Through AI.]

When I was finishing my PhD thesis, there came an important moment where I had to pick an inspirational quote to put at the start. This was a big deal to me at the time — impressive books often carry impressive quotes, so aping the practice felt like being a proper intellectual. After a little thinking, though, I got it into my head that the really clever thing to do would be to subvert the concept and pick something silly. So I ended up using this from The House At Pooh Corner by A. A. Milne:

When you are a Bear of Very Little Brain, and you Think of Things, you find sometimes that a Thing which seemed very Thingish inside you is quite different when it gets out into the open and has other people looking at it.

It felt very relevant, in the thick of thesis writing, that the quote was both self-deprecating and about the difficulty of effective communication. And actually, the more I thought about it, the more I realised I hadn’t picked something silly after all. Rather, this passage brilliantly expresses a profound fact about meaning[1].

The illusion of transparency

We’ve all had that feeling, I think, of not being able to find the right word — that sense of grasping for something just out of reach. Of looking for the exact term that will capture the Thing that’s so Thingish in our mind and communicate it with complete fidelity. Yet, often, even when we find what we thought we wanted, it still doesn’t quite do the trick.

The fundamental problem is that language is a compression of whatever is going on in our minds. Each word or phrase is located in a personal web of meaning, which is high dimensional and unique to ourselves. So when we say something, we never actually pass on the full meaning we intend. There’s a whole world of context that stays put in our brain, getting approximated away in our public statement.

This is a ubiquitous source of communication failures. I’ve heard it referred to before as the

‘illusion of transparency’

:

The illusion of transparency is the misleading impression that your words convey more to others than they really do. Words are a means of communication, but they don't in themselves contain meaning. The word apple is just five letters, two syllables. I use it to refer to a concept and its associations in my mind, under the reasonable assumption that it refers to a similar concept and g

...
If we get things right, AI could have huge benefits
by Algon — 2025-06-26T08:19:24.321Z

There is no contradiction between AI carrying huge potential risks, and it carrying huge potential upsides if we navigate the risks. Both are a consequence of the prospect of AI becoming extremely powerful.

The benefits that human-aligned AGI could bring are a major part of what motivates researchers to build such systems. Things we can confidently expect based on extrapolating from current technology include:

The development of more advanced technology in fields like agriculture, medicine, energy, computing, and transportation.
Economic growth in the form of widespread access to services that are currently provided by remote workers.
Robots doing much of our manual labor.

Examples that are more speculative, but seem physically feasible, include:

Ending diseases.
Curing aging and involuntary death.
Exploring and colonizing the galaxy.
Growing the economy by many orders of magnitude and ending poverty as we know it.

But to get these benefits, we’d have to ensure not only that we use these systems for good instead of for ill, but that they can be human-aligned at all. We’d have to get our act together, but many experts worry that we’re on a path to failure.

Related

What would a good future with AGI look like?

What are the potential benefits of advanced AI?

What is "transformative AI"?
Advanced AI is a big deal even if we don’t lose control
by Algon — 2025-06-26T08:19:12.976Z

Context: This is a linkpost for https://aisafety.info/questions/NM3G/10:-Advanced-AI-is-a-big-deal-even-if-we-don%E2%80%99t-lose-control

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

So far, we’ve discussed one class of possible consequences of advanced AI: systems ending up with different values than we intended, and autonomously deciding to seize control. This is the main concern that this site is about.

But of course, introducing a highly powerful class of tools into the world, let alone a new intelligent species, will have many other kinds of implications:

Even today’s systems are already quite useful, and at the same time cause significant harms. For example, art-generating AI is generating fun art for consumers, but is also making it harder for artists to find employment. Future AI that substitutes for humans in more sectors might create much more wealth and also cause much more wide-ranging economic upheaval.
Future advances may result in catastrophic risks from deliberate misuse of AI. Misuse is a separate problem from loss of control. For example, bad actors could use sufficiently powerful AI to design bioweapons or hack computer systems. Instead of the complicated research process that’s required now, AI could allow people to do these things at the press of a button.
Advanced AI has important political implications. Such a powerful tool could result in power being concentrated in a narrow set of people controlling it, like AI companies. It could also be used by governments to control their populations, and could upset the current balance of power, perhaps resulting in tensions and even war surrounding its development. Eventually, it could be used to create a permanent totalitarianism or inadvertently lock in some set of moral values we might not have wanted to.
If AI systems attain consciousness or personhood, that carries moral implications about how we should treat them. We may want to ensure we’re nice to them so they’ll be nice to us, or because we value their welfare, or because it makes us better people. Their uninterpretable nature, and our lack of understanding of concepts around consciousness, make it especially hard to navigate these implications.
When AI becomes as inventive as humans or more, we may see a drastic spee

...
Defeat may be irreversibly catastrophic
by Vishakha — 2025-06-26T08:19:00.543Z

Context: This is a linkpost for https://aisafety.info/questions/NM3P/9:-Defeat-may-be-irreversibly-catastrophic

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

When you imagine a global catastrophe, maybe the kind of event that comes to mind is one that strikes a big blow against human civilization but doesn’t ruin it permanently. Events like pandemics, environmental disasters, and even nuclear wars would cause enormous suffering and millions or billions of deaths, but would probably leave part of human civilization alive to recover. Future generations could still flourish.

An AI takeover catastrophe is not like that. An inherent feature of AI taking over control for its own long-term purposes is that it retains that control permanently. We’d be facing an adversary that, rather than striking a one-off blow, continues to optimize the world for its own ends long-term.

Such a scenario could result in full human extinction, because:

The AI may kill us deliberately, because we’re a threat. For example, we might build a different powerful AI that could compete with it. (In the later stages of an AI takeover, maybe humanity will be not even a threat — but then again, it will then also be extremely easy to kill us.)
The conditions supporting human life are unlikely to persist when there’s a technologically hyper-advanced AI civilization using the world for its own ends. Any of a wide range of changes such a civilization will plausibly make to the world — capturing all the sun’s energy output, changing the composition of the atmosphere, using the Earth’s surface for building materials — would result in our extinction

More generally, the risk of permanent loss of the potential for good outcomes for humanity is called existential risk. Human extinction is the main and most straightforward case. But in a world controlled by misaligned AI, even if humans do stay alive, it’s likely that our future would be much worse than we could have hoped for.

Nor could we expect a misaligned AI itself to attain the type of civilization we could have hoped to build ourselves — it will build big and complicated projects, but values like joy and consciousness are unlikely to be instrumentally useful to those projects. It would be a coincidence if they ended up leading what we ...
If Not Now, When?
by Yair Halberstadt — 2025-06-26T06:10:20.375Z

Hillel... would say: "If I am not for me, who will be for me? And If I am just for myself, what am I? And if not now, when?"

Pirkei Avot 1,14

That last question has usually been interpreted as a rhetorical one. If you're not going to do it now, you'll never get around to it!

But this is clearly a bad principle to live by. If you start cooking too early, the food will be cold when you want to eat it. If you pick a fruit before it's ripe it will be sour. When you are exhausted the quality of your work goes down. And when you have many things to do, you have to prioritise which comes first.

There is a season and a time for everything...

Ecclesiastes 3,1

Perhaps we would do better interpreting it literally. It's fine to push things off, but first you must answer the question: "When?". At what point in time, or under which change in circumstances, will you take this action? If you have a good answer to this, all is fine, but if you don't it's likely you are just pointlessly procrastinating, and the job will never be done.
How Much Data From a Sequencing Run?
by jefftk — 2025-06-26T02:30:06.160Z

Cross-posted from my NAO
Notebook.

Manufacturers often give optimistic estimates for how much data their
systems produce, but performance in practice can be pretty different.
What have we seen at the NAO?

We've worked with two main sample types, wastewater and nasal swabs,
but Simon already wrote
something up on swabs so I'm just going to look at wastewater
here.

We've sequenced with both Illumina and Oxford Nanopore (ONT), and the
two main flow cells we've used are the:

Illumina
NovaSeq X 25B, at 2x150. List price is $16,480, though we've seen
groups negotiate discounts in the range of 15%. Marketed
as producing a maximum of "52 billion single reads" (26B read pairs,
8,000 Gbp).

ONT
PromethION. List price is $900 in bulk, compared to $600 for the smaller
MinION flow cell. Since the PromethION flow cells have 5x the pores
(2,675 vs 512) and correspondingly higher output, I think most folks
who are using MinION should be considering switching over to the
PromethION. Marketed
as producing ">200 Gb on metagenomic samples".

Ignoring library prep consumables, labor, lab space, machine cost, and
looking just at the flow cells, if we took the manufacturer's costs at
face value this would be:

Illumina 25B: $2.06/Gbp ($16,480 / 8,000 Gbp)

ONT PromethION: $4.50/Gbp ($900 / 200 Gbp)

With 25B flow cells we've generally seen output meeting or exceeding
the advertised 26B read pairs (8,000 Gbp). In our sequencing at BCL we've averaged 29.4B
read pairs per sample (8,800 Gbp; n=37), while recently MU has been averaging
27.2B read pairs (8,200 Gbp; n=4, older runs were ~20% lower). It's
great to be getting more than we expected!

On the other hand, with PromethION flow cells we've generally seen
just 3.3 Gbp (n=25) on wastewater. This is slightly higher than the
2.5
Gbp we've seen with nasal swabs, but still far below 200 Gbp. We
don't know why our output is so much lower than advertised, but this
is what we've seen so far.

This would give us:

Illumina 25B: $1.87/Gbp ($16,480 / 8,800 Gbp)

ONT PromethION: $272/Gbp ($900 / 3.3 Gbp)

We're still not done, though, because while this is correct in terms
of raw bases coming off the sequencer, with paired-end sequencing on
short fragments like we have in wastewater a portion of many of your
reads will be adapters.
We see a median of 170bp after adapter trim...
The Practical Value of Flawed Models: A Response to titotal’s AI 2027 Critique
by Michelle_Ma — 2025-06-25T22:15:27.172Z

Crossposted from my Substack.

@titotal recently posted an in-depth critique of AI 2027. I'm a fan of his work, and this post was, as expected, phenomenal*.

Much of the critique targets the unjustified weirdness of the superexponential time horizon growth curve that underpins the AI 2027 forecast. During my own quick excursion into the Timelines simulation code, I set the probability of superexponential growth to ~0 because, yeah, it seemed pretty sus. But I didn’t catch (or write about) the full extent of its weirdness, nor did I identify a bunch of other issues titotal outlines in detail. For example:

The AI 2027 authors assign ~40% probability to a “superexponential” time horizon growth curve that shoots to infinity in a few years, regardless of your starting point.
The RE-Bench logistic curve (major part of their second methodology) is never actually used during the simulation. As a result, the simulated saturation timing diverges significantly from what their curve fitting suggests.
The curve they’ve been showing to the public doesn’t match the one actually used in their simulations.

…And more. He’s also been in communication with the AI 2027 authors, and Eli Lifland recently released an updated model that improves on some of the identified issues. Highly recommend reading the whole thing!

That said—it is phenomenal*, with an asterisk. It is phenomenal in the sense of being a detailed, thoughtful, and in-depth investigation, which I greatly appreciate since AI discourse sometimes sounds like: “a-and then it’ll be super smart, like ten-thousand-times smart, and smart things can do, like, anything!!!” So the nitty-gritty analysis is a breath of fresh air.

But while titotal is appropriately uncertain about the technical timelines, he seems way more confident in his philosophical conclusions. At the end of the post, he emphatically concludes that forecasts like AI 2027 shouldn’t be popularized because, in practice, they influence people to make serious life decisions based on what amounts to a shoddy toy model stapled to a sci-fi short story (problematically disguised as rigorous, empirically-backed research).

He instead encourages plans that are “robust to extreme uncertainty” about AI and warns against taking actions that “could blow up in your face” if you're badly wrong.

On the face of it, that sounds reasonable enough. But is it?

“Robust” Planning: The Peopl

...