trentmkelly/LessWrong-43k huggingface dataset:
Forging A New AGI Social Contract
by Deric Cheng — 2025-04-10T13:41:11.817Z
Link: https://agisocialcontract.substack.com/p/forging-a-new-agi-social-contract

This is the introductory piece for a series of essays written by AI economists, policy researchers, and political thinkers on the topic of a new AGI Social Contract.

Current contributors include:

Anton Korinek

,

Deger Turan

(CEO of Metaculus),

Steve Omohundro

,

Iason Gabriel

(DeepMind),

Julian Jacobs

(DeepMind),

Sam Manning

(GovAI),

Dylan Hadfield

(MIT),

Seth Lazar

(ANU),

Peter Salib

(UH),

Colleen McKenzie

(AOI),

Philip Tomei

(AOI),

Dean Ball

(Hyperdimensional),

Justin Bullock

(ARI), Anna Yelizarova (Windfall Trust).

If you're interested in getting updates on this anthology, leave us your email on our Substack at this link here.

“Digital technologies will bring the world into an era of more wealth and abundance and less drudgery and toil. But there’s no guarantee that everyone will share in the bounty. The outcome—shared prosperity or increasing inequality—will be determined not by technologies but by the choices we make as individuals, organizations, and societies.”

Erik Brynjolfsson - The Great Decoupling, 2015

Society has long operated on a simple bargain: we contribute our labor and, in return, gain income, security, and a stake in the economy. Governments tax our wages to fund public services; corporations rely on human workers to create value; and in return, workers expect their efforts to be rewarded with opportunity and security. This centuries-old implicit bargain will soon come under strain.

There is significant evidence to suggest that artificial general intelligence (AGI) systems capable of directly replacing much of human labor are right around the corner.

Leading

AI

researchers

are not just predicting that these systems will be better at performing tasks than today’s AI - they are suggesting that human intelligence will no longer be unique. Some suggest that we are on the threshold of a major inflection point akin to the

Agricultural or Industrial Revolutions

.

If these claims turn out to be true, they will force us to completely reevaluate how we organize labor throughout the economy. Historically, technology has mainly increased the productivity of workers, automating repetitive tasks and increasing the efficiency of human effort. In contrast, AGI represents something fundamentally different – a technology capable of exceeding human perf...
Why Experienced Professionals Fail to Land High-Impact Roles (FBB #5)
by gergogaspar — 2025-04-10T12:46:35.862Z

Crossposted on the EA forum and Substack.

Introduction

There are many reasons why people fail to land a high-impact role. They might lack the skills, don’t have a polished CV, don’t articulate their thoughts well in applications[1] or interviews, or don't manage their time effectively during work tests. This post is not about these issues. It’s about what I see as the least obvious reason why one might get rejected relatively early in the hiring process, despite having the right skill set and ticking most of the other boxes mentioned above. The reason for this is what I call context, or rather, lack thereof.

On professionals looking for jobs

It’s widely agreed upon that we need more experienced professionals in the community, but we are not doing a good job of accommodating them once they make the difficult and admirable decision to try transitioning to AI Safety.

Let’s paint a basic picture that I understand many experienced professionals are going through, or at least the dozens I talked to at EAGx conferences.

They do an AI Safety intro course
They decide to pivot their career
They start applying for highly selective jobs, including ones at OpenPhilanthropy
They get rejected relatively early in the hiring process, including for more junior roles compared to their work experience
They don’t get any feedback
They are confused as to why and start questioning whether they can contribute to AI Safety

If you find yourself continuously making it to later rounds of the hiring process, I think you will eventually land the job sooner or later. The competition is tight, so please be patient! To a lesser extent, this will apply to roles outside of AI Safety, especially to those aiming to reduce global catastrophic risks.

But for those struggling to penetrate later rounds of the hiring process, I want to suggest a potential consideration. Assuming you already have the right skillset for a given role, it might be that you fail to communicate your cultural fit and contextual understanding of AI Safety to hiring managers, or you still lack those for now.

This isn’t just on you, but on all of us as a community. In this article, I will outline some ways that job seekers can better navigate the job market, but focus less on how the community can avoid altruistic talent getting lost on the journey. That is worth its own forum post!

To make it clear again...
Linkpost to a Summary of "Imagining and building wise machines: The centrality of AI metacognition" by Johnson, Karimi, Bengio, et al.
by Chris_Leong — 2025-04-10T11:54:37.484Z

How does my summary compare to the original paper?

Obviously, the summary is shorter :-).

Additionally, I have:

• Streamlined the user journey with collapsable sections

• Communicated some ideas in an easier to digest format

• Added commentary and a glossary

Whilst I've tried as hard as possible to represent the views in the original (including making some updates in response to feedback from one of the authors), I can't guarantee perfect accuracy.

Original paper Summary

Authors of the Paper:

Samuel G. B. Johnson, Amir-Hossein Karimi, Yoshua Bengio, Nick Chater, Tobias Gerstenberg, Kate Larson, Sydney Levine, Melanie Mitchell, Iyad Rahwan, Bernhard Schölkopf, Igor Grossmann

Abstract of Summarised Paper:

Recent advances in artificial intelligence (AI) have produced systems capable of increasingly sophisticated performance on cognitive tasks. However, AI systems still struggle in critical ways: unpredictable and novel environments (robustness), lack of transparency in their reasoning (explainability), challenges in communication and commitment (cooperation), and risks due to potential harmful actions (safety). We argue that these shortcomings stem from one overarching failure: AI systems lack wisdom.

Drawing from cognitive and social sciences, we define wisdom as the ability to navigate intractable problems - those that are ambiguous, radically uncertain, novel, chaotic, or computationally explosive - through effective task-level and metacognitive strategies. While AI research has focused on task-level strategies, metacognition - the ability to reflect on and regulate one's thought processes - is underdeveloped in AI systems. In humans, metacognitive strategies such as recognizing the limits of one's knowledge, considering diverse perspectives, and adapting to context are essential for wise decision-making. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety.

By focusing on developing wise AI, we suggest an alternative to aligning AI with specific human values - a task fraught with conceptual and practical difficulties. Instead, wise AI systems can thoughtfully navigate complex situations, account for diverse human values, and avoid harmful actions. We discuss potential approaches to building wise AI, including benchmarking metacognitive abilities and training AI sys...
Grounded Ghosts in the Machine - Friston Blankets, Mirror Neurons, and the Quest for Cooperative AI
by Davidmanheim — 2025-04-10T10:15:54.880Z
Link: https://davidmanheim.com/exploring-cooperation-11

This is another post in my ongoing "Exploring Cooperation" substack series, focused on something more directly related to LLMs and alignment - I am including the post in its entirety.
Throughout this series, we’ve repeatedly circled around the requirements for genuine cooperation—shared context, aligned goals, and outcomes that matter to the participants. In earlier posts, we explored the importance of preferences and identity, noting that cooperation depends not just on behavior, but on agents who care about how things turn out and persist long enough for reciprocity and coordination to make sense. We also discussed why status and power can undermine cooperation, especially when incentives diverge or when agents lack continuity across time. Now, after laying this conceptual groundwork across discussions of evolution, economics, and history, we’re finally ready to turn toward the near term future—specifically, to ask what it would even mean to cooperate with AI systems. We'll first address LLMs lacking semantic participation, memory, and grounding, then move to newer and increasingly agentic LLM-based systems, and future AI.

Does it matter if LLMs can’t “really” cooperate?

Large language models like ChatGPT are session-based systems—they do not persist identity or memory across interactions unless explicitly designed to do so, and they lack direct experience or embodiment in the world. Because of this, they arguably lack both linguistic and communicative grounding: LLMs are a form of (very smart) stochastic parrots, and there’s no stable context or stake in the outcome for the model itself. As a result, when LLMs appear to cooperate, they are essentially simulating cooperative behavior based on patterns in their training data, rather than participating in cooperation as an agent with preferences or goals. As I’ve presented before, current systems are just pretending. And this raises a critical question: if a system only mimics the surface-level structure of cooperation without being invested in outcomes, is it really cooperating?

And whether it "really" matters if LLMs are genuinely cooperating or just simulating it depends on what we’re trying to achieve and what we think cooperation requires. On one hand, if the outputs of simulated cooperation are indistinguishable from genuine cooperation—if tasks get done, goals are advanced, and humans feel understood—then...
New Paper: Infra-Bayesian Decision-Estimation Theory
by Vanessa Kosoy — 2025-04-10T09:17:38.966Z
Link: https://arxiv.org/abs/2504.06820

Diffractor is the first author of this paper.
Official title: "Regret Bounds for Robust Online Decision Making"

Abstract: We propose a framework which generalizes "decision making with structured observations" by allowing robust (i.e. multivalued) models. In this framework, each model associates each decision with a convex set of probability distributions over outcomes. Nature can choose distributions out of this set in an arbitrary (adversarial) manner, that can be nonoblivious and depend on past history. The resulting framework offers much greater generality than classical bandits and reinforcement learning, since the realizability assumption becomes much weaker and more realistic. We then derive a theory of regret bounds for this framework. Although our lower and upper bounds are not tight, they are sufficient to fully characterize power-law learnability. We demonstrate this theory in two special cases: robust linear bandits and tabular robust online reinforcement learning. In both cases, we derive regret bounds that improve state-of-the-art (except that we do not address computational efficiency).

In our new paper, we generalize Foster et al's theory of "decision-estimation coefficients" to the "robust" (infa-Bayesian) setting. The former is the most general known theory of regret bounds for multi-armed bandits and reinforcement learning, which comes close to giving tight bounds for all "reasonable" hypothesis classes. In our work, we get an analogous theory, even though our bounds are not quite as tight.

Remarkably, the result also establishes a tight connection between infra-Bayesianism and Garrabrant induction. Specifically, the algorithm which demonstrates the upper bound works by computing beliefs in a Garrabrant-induction-like manner[1], and then acting on these beliefs via an appropriate trade-off between infra-Bayesian exploitation and exploration (defined using the "decision-estimation" approach).

It seems quite encouraging that the two different theories which came out of thinking about "logical uncertainty" (infra-Bayesianism and Garrabrant induction) can be unified in this manner[2], boosting our confidence that we are on the right path.

The paper assumes no prior familiarity with either infra-Bayesianism or Garrabrant induction.

^

In the start of each episode.

^

Although we think that a fuller treatment of logical uncertainty requires introducing meta

...
Electric Lunchbox
by jefftk — 2025-04-10T02:40:07.412Z

People are often a lot more interested in hot meals, and my kids are
no exception. I've tried a bunch of options here including

putting rocks in
thermoses (turns out kindergarteners worry more than you might
think about whether they'll accidentally eat rocks that are bigger
than their mouths), bringing a

microwave
and

toaster
(good, but too bulky for school especially when you count the
battery), and ramen (great, but Lily only likes one kind and I'm
worried she'll get sick of it). We recently got an electric lunchbox (

this one because it
was on sale, but there are a bunch) and it's pretty great!

It's insulated, and we prepare it the night before and put it in the
fridge:

In the morning I set the timer:

It only goes up to 4hr, perhaps for danger
zone reasons, but since the kids leave at 8am and lunch is at
12:30 that's just right (4hr timer, 30min heating).

When the timer gets to zero it starts heating, and counts down from
30min (by default; adjustable):

Tastes marginally better than microwave food, and loads better than
cold food.

I haven't yet tested if it's powerful enough to handle frozen food. I
hope it is, since then you could prepare it whenever and leave it in
the freezer
until ready to eat.

I also wonder if it could bake bread. It only goes up to 90C and
it be a bit like steamed bread,
but freshness
might outweigh texture here.
Scoping LLMs
by erik — 2025-04-10T00:32:13.771Z

Emile Delcourt, David Baek, Adriano Hernandez, Erik Nordby with advising from Apart Lab Studio

Introduction & Problem Statement

Helpful, Harmless, and Honest (”HHH”, Askell 2021) is a framework for aligning large language models (LLMs) with human values and expectations. In this context, "helpful" means the model strives to assist users in achieving their legitimate goals, providing relevant information and useful responses. "Harmless" refers to avoiding generating content that could cause damage, such as instructions for illegal activities, harmful misinformation, or content that perpetuates bias. "Honest" emphasizes transparency about the model's limitations and uncertainties—acknowledging when it doesn't know something, avoiding fabricated information, and clearly distinguishing between facts and opinions. This framework serves as both a design principle for AI developers and an evaluation criterion for assessing how well LLMs balance being maximally useful to users while minimizing potential harms and maintaining truthfulness in their outputs.

However, since computation in a transformer considers every part of the input prompts’ “context window,”[1] generative AI deployments (applications) struggle to apply any effective limits beyond “egregious” safety/toxicity backstops (the few things for which there is universal agreement, Bengio et al 2024, Ji et al 2023, Buyl et al 2025, Kumar et al 2021). Any fragment of text can influence attention enough to supersede earlier instructions, subverting the purposes of the application (see “state of jailbreak” section below). Typically, a preamble is designated as a “system prompt” with enforcement expectations, but applications still cannot establish any guarantees to prevent abuse in open-ended user interactions. Even when foundation models are intentionally trained to be helpful, harmless, and honest, we argue that responsible AI systems must also “stay in their lane” - and need to also be “Honed” (HHH+H).

In this article, we address how important HHH+H is, specifically to achieve effective scoping boundaries, why solutions have not achieved it, and the requirements to get there.

While every organization building or using a technology to support their goals typically makes the minimum viable capabilities clear (which LLM capability advancements have all but solved), those organizations also have explicit and ...
How familiar is the Lesswrong community as a whole with the concept of Reward-modelling?
by Oxidize — 2025-04-09T23:33:18.044Z

I initially assumed that the concept of reward-modelling would be something most Lesswrongers were very familiar with. After all, this is one of the best communities for conversing on the topic. And a large percentage of all posts are AI or doom-based

However, my skeptical side quickly kicked in and I started to doubt my initial assumption. As I realized that my assumption could be confirmation bias. I am personally highly invested in reward-modelling, and I tend to ignore information that has little to no relation to it. Additionally, I do not have access to any actual data on this, nor have I considered perspectives outside of my own.

Much of my beliefs around the concept of agency and reward-modelling are well modeled by the channel RobertMilesAI. How familiar is the community with the concepts expressed in this channel?
How aware is the community as a whole of the concept?
How interested is the community as a whole in the concept?
I would be very thankful for any replies. As I'm very invested in the concept of reward-modelling. So any outside perspectives on the topic are very valuable to me.
What can we learn from expert AGI forecasts?
by Benjamin_Todd — 2025-04-09T21:34:53.148Z
Link: https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/

An introductory article on AI forecasts. Posted for feedback.

As a non-expert, it would be great if there were experts who could tell us when we should expect artificial general intelligence (AGI) to arrive.

Unfortunately, there aren’t.

There are only different groups of experts with different weaknesses.

This article is an overview of what five different types of experts say about when we’ll reach AGI, and what we can learn from them (that feeds into my full article on forecasting AI).

In short:

Every group shortened their estimates in recent years.

AGI before 2030 seems within the range of expert opinion, even if many disagree.

None of the forecasts seem especially reliable, so they neither rule in nor rule out AGI arriving soon.

In four years, the mean estimate on Metaculus for when AGI will be developed has plummeted from 50 years to 5. There are problems with the definition used, but the graph reflects a broader pattern of declining estimates.

Here’s an overview of the five groups:

AI experts

1. Leaders of AI companies

The leaders of AI companies are saying that AGI arrives in 2–5 years, and appear to have recently shortened their estimates.

This is easy to dismiss. This group is obviously selected to be bullish on AI and wants to hype their own work and raise funding.

However, I don’t think their views should be totally discounted. They’re the people with the most visibility into the capabilities of next-generation systems, and the most knowledge of the technology.

And they’ve also been among the most right about recent progress, even if they’ve been too optimistic.

Most likely, progress will be slower than they expect, but maybe only by a few years.

2. AI researchers in general

One way to reduce selection effects is to look at a wider group of AI researchers than those working on AGI directly, including in academia. This is what Katja Grace did with a survey of thousands of recent AI publication authors.

The survey asked for forecasts of “high-level machine intelligence,” defined as when AI can accomplish every task better or more cheaply than humans. The median estimate was a 25% chance in the early 2030s and 50% by 2047 — with some giving answers in the next few years and others hundreds of years in the future.

The median estimate of the chance of an AI being able to do the job of an AI researcher by 2033 was 5%.1

They were also asked about when they expected AI could ...
Thoughts on AI 2027
by Max Harms — 2025-04-09T21:26:23.926Z
Link: https://intelligence.org/2025/04/09/thoughts-on-ai-2027/

This is part of the MIRI Single Author Series. Pieces in this series represent the beliefs and opinions of their named authors, and do not claim to speak for all of MIRI.

Okay, I'm annoyed at people covering AI 2027 burying the lede, so I'm going to try not to do that. The authors predict a strong chance that all humans will be (effectively) dead in 6 years, and this agrees with my best guess about the future. (My modal timeline has loss of control of Earth mostly happening in 2028, rather than late 2027, but nitpicking at that scale hardly matters.) Their timeline to transformative AI also seems pretty close to the perspective of frontier lab CEO's (at least Dario Amodei, and probably Sam Altman) and the aggregate market opinion of both Metaculus and Manifold!

If you look on those market platforms you get graphs like this:

Both of these graphs have a peak (a.k.a. mode) in or at the very tail end of 2027 and a weighted midpoint (a.k.a. median) in 2031. Compare with this graph from AI 2027's timelines forecast:

Notably, this is apples-to-oranges, in that superhuman coders are not the same as "AGI." But if we ignore that, we can hallucinate that FutureSearch has approximately the market timelines, while Eli Lifland (and perhaps other authors) has a faster, but not radically faster timeline. The markets are less convinced that superintelligence will follow immediately thereafter, but so are the authors of AI 2027! Predictions are hard, especially about the future, etc.

But I also feel like emphasizing two big points about these overall timelines:

Mode ≠ Median

As far as I know, nobody associated with AI 2027, as far as I can tell, is actually expecting things to go as fast as depicted. Rather, this is meant to be a story about how things could plausibly go fast. The explicit methodology of the project was "let's go step-by-step and imagine the most plausible next-step." If you've ever done a major project (especially one that involves building or renovating something, like a software project or a bike shed), you'll be familiar with how this is often wildly out of touch with reality. Specifically, it gives you the planning fallacy.

In the real world, shit happens. Trump imposes insane tariffs on GPUs. A major terrorist attack happens. A superpower starts a war with its neighbor (*coughTaiwancough*). A critical CEO is found to be a massive criminal. Congress passes insan...