trentmkelly/LessWrong-43k huggingface dataset:
Can LLM-based models do model-based planning? 
by jylin04 — 2025-04-16T12:38:00.793Z
Link: https://docs.google.com/document/d/1eYkXycZEu93KOdwh6sOm0hHXr-yZ61lRivWtL5yohRc/edit?usp=sharing

I recently spent a few months thinking about whether LLM-based models can do model-based planning, and wrote a ~40-page report on it: "Report on LLMs and model-based planning". The doc is a bit rough around the edges still - most notably, the concepts of "efficient planning" and "sufficiently convoluted" tasks in section 1 are incompletely defined - but I thought I would share it in the current form, in case others could find the framework or early conclusions useful.

The summary is as follows:

In this report, I investigate the question of whether current LLM-based models have the cognitive capability of model-based planning (MBP).

As motivation, model-based planning is a frequently mentioned crux in the ongoing debate about whether LLM-based models are on track to scale to AGI. However, the debate is frequently underspecified, with different people taking e.g. “world-model” to mean different things. In this report, l first operationalize model-based planning in a way that in principle can be compared against information-processing patterns in trained LLMs, and explain why I think the operationalization may be necessary for AIs to achieve certain consequential tasks in the real world. I then explore whether current LLM-based models can do it, based mainly on their performance on related benchmarks.

My main findings are that

Whether or not LLM-based models can do model-based planning may be bottlenecked by the complexity of the states required by the world-model: in particular whether or not they can be compressed into a compact representation in token form.

Current “pure LLMs” through GPT-4 probably cannot do model-based planning as defined here for a nontrivial number of planning steps, over world-models with even very simple states, based on their performance on existing benchmarks.

Current reasoning models through o1 probably cannot do model-based planning as defined here for a nontrivial number of planning steps, over world-models with relatively simple states, based on their preliminary performance on existing benchmarks.

Existing benchmarks may imperfectly track model-based planning under this operationalization, and I suggest an idea for a new benchmark to fill in the gaps.

Looking ahead, I think LLM-based architectures at scale could plausibly support model-based planning inefficiently, with one pote

...
The road from human-level to superintelligent AI may be short
by Vishakha — 2025-04-16T08:35:54.376Z
Link: https://aisafety.info/questions/NM38/4:-The-road-from-human-level-to-superintelligent-AI-may-be-short

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

It’s tempting to think that, even if AI can become vastly smarter than humans, getting to that point will take a vast amount of time, like centuries. However:

As discussed before, inputs to AI progress — including investment, hardware quality, and software efficiency — are growing exponentially. Exponential growth can be deceptively fast. AI that fully substitutes for humans would bring returns on the scale of the whole economy, as well as crucial geopolitical advantages. And the returns from superintelligence would be on the scale of a much bigger future economy. So we can expect efforts to develop these technologies to become correspondingly huge, helping drive rapid exponential growth.
Just like things in the universe come in an extremely wide range of sizes, and different humans occupy only a small window of that range, maybe the range of human intelligence is also just a small part of the range of possible intelligences. If so, it’s likely that AI would zoom past that window quickly. In Go and Chess, it took just years between AI matching the best human players and AI becoming far stronger than the best human players — even so much stronger that it has no more use for human input.
Future AI progress might come as a result of genuinely new paradigms and insights, rather than the smooth scaling with increasing resources that we’ve been seeing. Sometimes, finding a way to tap into a new kind of phenomenon leads to sudden huge leaps in output — consider bombs before and after the Manhattan Project.

But most importantly:

AI systems will increasingly contribute to AI progress themselves. That could look like a single AI recursively self-improving, writing smarter versions of itself and using its new smarts each time to write the next version. But more indirectly, an economy in which new agents with human-like intelligence can be created, sped up, and copied will operate at faster speeds than ours. Such an economy may see explosive growth as a result of feedback loops such as companies using AI to make money and spending that money on further AI improvements.

We don’t know how long after human-level A...
Human-level is not the limit
by Vishakha — 2025-04-16T08:33:15.498Z
Link: https://aisafety.info/questions/NM3A/3:-Human-level-is-not-the-limit

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

We’ve built technologies like skyscrapers that are much larger than us, bulldozers that are much stronger, and airplanes that are much faster. Similarly, there are good reasons to think that machines can eventually become much more capable at general cognitive problem-solving.

On an abstract level, our intelligence came from evolution, and while evolution results in well-optimized systems, it can’t “think ahead” or deliberately plan design choices. Humans are just the first generally intelligent system that worked, and there’s no reason to expect us to be close to the most effective design. Moreover, evolution works under heavy constraints that don’t affect the creators of AI.

Advantages that AI “brains” can eventually gain over ours include:

Sheer size. The size of human heads is limited by the constraint of them having to fit through the birth canal, but there’s nothing stopping an AI from being, say, the size of a warehouse. And we don’t know how many computations a human brain does, but most estimates imply that it’s a lot less than a current datacenter, let alone one built with future technology.
Greater serial speed. Signals in a digital computer can propagate at millions of times the speed they do in a human brain. That means a computer could do the same computation in a millionth of the time, as long as it applied a million times the computing power per second. To a mind like that, we’d be almost frozen in time.
The ability to replicate themselves as easily as copying some files on a computer. As a result of having shared structure and common knowledge of that structure, AIs could coordinate and share information and skills much more easily than human individuals. (There’s a blurry line between systems being very well-coordinated and being one large system — see the first point.) And there could be many of them as soon as they were created. In a science fiction story, a genius inventor might design one robot, and then have one robot. But AI in the real world is not like that — the computing power used to create (train) a single model can run many instances of that model.
The ability to self-

...
AI may attain human-level soon
by Vishakha — 2025-04-16T08:28:55.592Z
Link: https://aisafety.info/questions/NM3C/2:-AI-may-attain-human-level-soon

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

The main companies building AI, such as OpenAI, Anthropic, and Google DeepMind, are explicitly aiming for artificial general intelligence: AI that has the same kind of general reasoning ability that humans have, and can do all the same tasks and jobs (at least the ones that don’t require a physical body).

We’re not quite there yet, and predicting technological progress is hard. But there are a few reasons to think we'll see human-level AI soon — in decades or maybe even just a few years.

Extrapolating intuitively from past progress, if AI took just a few years to go from a high-school-level to a PhD-level ability to answer questions in science, mathematics, and programming,[1][2][3] then it’s not crazy to think it could overcome the remaining hurdles to general human-level reasoning in a few more years.
Looking at the gap that remains between current and human-level AI, there isn't widespread agreement on a clear and qualitative factor that is missing. Progress doesn’t seem to be stalling out, and we can’t rule out the “scaling hypothesis”, which says that just putting more computation into training systems similar to today’s will continue to result in smarter AIs. Even if scaling isn’t enough, and we need some new ideas, it may not take many new ideas to successfully apply today’s general learning methods to build something human level.
Some people have tried to build quantitative models of future AI progress. One way of doing this is to estimate the level of computing power it would take to train a human-level model — for example, by human biological equivalents. We have major uncertainty about what that level is, but it’s probably not more than, say, ten orders of magnitude away, and we’re on track to continue making big leaps toward it. Large companies are spending hundreds of billions per year building datacenter infrastructure intended largely for future AI. As of early 2025, the “Stargate” collaboration plans to build AI technology with half a trillion dollars. And hardware and software keep getting not just larger in scale, but better. Modeling AI progress is not an exact science, but

...
AI is advancing fast
by Vishakha — 2025-04-16T08:17:06.055Z
Link: https://aisafety.info/questions/NM37/1:-AI-is-advancing-fast

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

AI has existed since the 1950s, but in the 2010s and especially 2020s, “deep learning” systems have made great strides in capability. The most prominent deep learning systems are “large language models” (LLMs) trained on completing human text, which end up being able to do a surprisingly wide range of tasks.

What's driving this progress? A few different factors are scaling up exponentially:[1]

The amount of money spent (2.6x growth per year), which is now on the order of a hundred million for the largest training runs.
The quality of computing hardware technology (1.35x growth per year), measured in computations available per dollar — see also “Moore’s Law”.
The quality of software algorithms (3x growth per year), measured according to how much computing power it takes to reach some given level of performance.
The amount of data used to train language models (3x growth per year).

The total computing power applied to AI increases exponentially based on the first two points, and once you include software improvement, what you might call the “effective computing power” increases at an even faster exponential rate.

This growth compounds quickly. The computing power used to train the largest models in 2024 was billions of times greater than in 2010[2] — and that’s before taking into account better software.

As a result of this scaling, AI has become smarter. Various AI systems can now:

Drive cars. (For real this time.)
Generate realistic images, speech, music, and even video from a short text prompt.
Operate robots in factories and on construction sites.
Beat the best humans at many board and video games.
Predict the structures of proteins.

Scaling has also made some AI systems more general. A single language model like GPT-4o, which powers ChatGPT, can:

Hold a conversation like a human.
Play chess at the level of an intermediate human player, just from having seen lots of game transcripts.[3]
Give sensible advice on a wide range of topics.
Tutor users competently.
Search the web for new information.
Translate between many languages, even if they haven’t been trained on the specific pair.[4]

To get a sense of the ...
How Logic "Really" Works: An Engineering Perspective
by Daniil Strizhov — 2025-04-16T05:34:09.443Z

When you hear "logic," what comes to mind? Probably strict rules, truth tables, or the classic "if A, then B." It all seems clear—until you try to apply it to real life, where things rarely fit so neatly.

But what if logic isn't just about formulas? What if, at its core, it's about something even simpler: noticing the difference between one thing and another?

Textbooks start with symbols: P → Q, ∀x, ∃y. But maybe real logic begins earlier—at the moment we first draw a line between "this" and "not this."

This article offers a look at logic from an engineering angle. The goal isn't to throw out what you know, but to add a new tool. Sometimes, even a small shift in perspective can make disagreements clearer and reasoning more robust.

From an engineering standpoint, this is like analyzing a bridge's aerodynamics without examining its foundation.

Symbol Legend: The Alphabet of Structural Thinking

Note:
The symbols are not the point—they are just a way to make invisible structure visible. The real value is in the clarity and robustness of your reasoning.

Before we dive in, a quick note: these symbols are used as signposts to help notice the structure beneath the surface. You don't need to memorize them—just refer back as needed.

Symbol	Name	Description	Role
↘	Distinction	The fundamental act of creating a boundary, separating 'this' from 'not-this'.	The initiator.
○	Form	That which is defined and bounded by a Distinction (↘).	The resulting element.
⇒	Transformation / Inference	An asserted link or valid transition between established Forms (○).	The connection/path.
∅	Void / Undefined	A gap where inference (⇒) fails due to unclear ↘ or ill-defined ○.	Error: missing base.
⟲	Loop	Circular reasoning; a ⇒ pattern returning to its start without progress.	Pattern: useless cycle.
✶	Goal	The intended target Form (○) or state of a reasoning process.	The target marker.
✘	Contradiction	A clash where Forms (○) or Transformations (⇒) are incompatible.	Error: internal conflict.

This table is here for quick reference. It helps map and debug the structure of thought as you read. Now, let's see where logic actually begins.

Where This Framework Shines

In precise, formalized problems (like mathematics or classical logic puzzles), standard inference rules (⇒) are usually enough: the boundaries and forms are already clear.

But in real-world, "chaotic" questions—where terms are vague, boundaries are fuzzy, and disagreem...
Opportunity to to learn more about AI Innovation & Security Policy 
by PolicyTakes — 2025-04-16T01:35:27.203Z

Note: Sharing this workshop since LW has a lot of people interested in testing their fit for AI Governance. Sorry if this too spammy for this forum, feel free to let me know if so.
If you're interested in a job tackling tough problems at the intersection of AI, innovation, and national security, consider applying to Horizon’s

AI Innovation & Security Policy Workshop

, taking place July 11-13 in Washington DC.

What do you get?

Through hands-on sessions, expert panels, and networking opportunities, participants will gain practical insights into policy careers and develop the skills needed to effectively bridge the gap between the technology and policy worlds.
Dive into the critical issues of AI innovation and security policy, including:

National security applications and implications of AI
Energy production and grid capacity for data centers
Global technological competition, export controls, and China’s AI ecosystem
Model testing and evaluation frameworks
The intersection of AI, cyber security, and critical infrastructure

Past Horizon workshop participants have said:

“The workshop was my first step into my new career trajectory and it was inspiring to hear from such amazing speakers and meet like-minded peers.”
“Horizon gave me the confidence to change my career and move to DC to work in government. I also gained a new network of friends and learned about emerging tech.”
“I doubt I would have made the transition into policy without the Horizon workshop.”

Who is it for?

This workshop is for US citizens from all backgrounds and experience levels.
We welcome technologists, researchers, industry experts, and other professionals who bring valuable domain knowledge but may be unfamiliar with policy processes or government.
No prior policy experience is required.

Speakers include (more to be announced):

Samuel Hammond – Chief Economist @ Foundation for American Innovation
Lennart Heim – AI and compute policy @ RAND
Neil Chilson – Head of AI Policy @ Abundance Institute, former CTO @ FTC
Ryan Arant – Director of the N7 Initiative @ Atlantic Council, former Director on NSC and NEC in Trump #1
Ella Guest – AI capability evaluations @ RAND
Satya Thallam – Senior Fellow @ Foundation for American Innovation, former OMB in Trump #1 and Senate HSGAC staff

Want to learn more? Check out the

workshop page

and appl...
D&D.Sci Tax Day: Adventurers and Assessments
by aphyer — 2025-04-15T23:43:14.733Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

Estimated Complexity: 3/5 (this is a guess, I will update based on feedback/seeing how the scenario goes)

STORY

It's that time of year again. The time when the Tithe Assessment Exactors demand that all adventurers pay taxes on the various monster parts they have hacked off and sold in the past year. And, more importantly for you, the time when clients begin banging on your door looking for advice on how to minimize their taxes.

This used to be a straightforward, if complex, application of the published tax rules. But ever since the disaster a few years back (when one of your clients managed to pay 1 silver piece in tax and then receive as a rebate several thousand gold plus the princess's hand in marriage) the Tithe Assessment Exactors have been reluctant to publish the exact tax rules they use.

So when an adventuring team retains your services to help with their taxes, this is going to be a bit more difficult than before. You don't have a list of the new tax rules: the one thing you do have is a dataset of the taxes that various adventurers have been charged under them.

Your clients have got a list for you of how many monster parts of each kind they sold in the past year - it's too late to change that. The one thing you can use is that, while their adventuring party has pooled its finances, each of them will be filing their taxes individually. Perhaps, by cleverly allocating the monster parts among them, you can minimize their overall tax burden compared to other adventurers (and win more business...or at least avoid getting your head knocked off by their hulking barbarian!)

DATA & OBJECTIVES

Your clients have sold the following monster parts in the past year:

4 Cockatrice Eyes
4 Dragon Heads
5 Lich Skulls
7 Unicorn Horns
8 Zombie Hands

They need to divide these monster parts among the four of them for tax purposes.

The parts assigned to all four adventurers combined must add up to the totals above.
So you could assign taxes for:

The 4 Cockatrice Eyes and 1 of the Lich Skulls to the 1st adventurer
The 4 Dragon Heads and 1 of the Unicorn Horns to the 2nd adventurer
2 Lich Skulls, 3 Unicorn Horns, and 4 Zombie Hands to each of the 3rd and 4th adve

...
Should AIs be Encouraged to Cooperate?
by PeterMcCluskey — 2025-04-15T21:57:06.096Z
Link: https://bayesianinvestor.com/blog/index.php/2025/04/15/should-ais-be-encouraged-to-cooperate/

I have an idea for a fairly cheap strategy for slightly reducing the
risk of catastrophic harm from AI.

This post was inspired by a recent LessWrong post How AI Takeover Might
Happen in 2
Years.

That post describes a scenario in which most humans die, mainly because
an AI that is first to become powerful enough to attempt world conquest
sees risks that other AIs, imperfectly aligned with it, will cause
danger soon. This creates a perceived need to engage in a few violent
pivotal
processes,
rather than using a combination of persuasion and negotiation.

The specific scenario in that post seems fairly unrealistic. I see less
than a 5% chance that a leading AI within the next 10 years will want to
attempt that kind of world conquest. But it's close enough to being
realistic that I want to analyze a class of scenarios that are similar
to it.

I want to nudge AIs in the direction of being more willing to cooperate
with each other, in order to reduce the risk of such a conquest.
Specifically, I want them to negotiate a peaceful outcome instead of
using pre-emptive destruction.

Prior Work

There has been a moderate amount of writing on the general topic of
cooperative
AI. A lot
of it treats AIs as generic agents, usually rational, who are influenced
in the ways that humans are influenced. I'm focusing here on the
neglected(?) approach of asking how we can and should influence the
predispositions of the first transformative AIs.

The main paper that's close to studying the kind of cooperation that I
want is Cultural Evolution of Cooperation among LLM
Agents, which uses the Donor Game to
show that Claude 3.5 cooperates much more with other LLMs than do Gemini
1.5 and GPT-4o.

Assumptions

To clarify the class of scenarios I'm imagining:

I'm assuming that the leading AI gets at most a weakly decisive
strategic advantage. I.e. when it thinks it might be able to succeed at
world conquest, other AIs are close enough to its capability level that
it can't predict that it will get a more decisive advantage by further
developing its capabilities. This is inconsistent with some fast
take-off models of AI, and with some models that attribute take-off to
some unique insight that's hard for other AI companies to replicate.
But it seems consistent with the kind of take-off I foresee from an
extrapolation of current trends.

I'm assuming that, at the stage at which it becomes dangerous, the AI
...