trentmkelly/LessWrong-43k huggingface dataset:
Forecast AI 2027
by ChristianWilliams — 2025-06-12T21:12:04.836Z
Link: https://www.metaculus.com/tournament/ai-2027/

AI 2027 is a forecasting scenario by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean. It predicts AI agents will begin automating AI research and development in 2027, triggering an intelligence explosion and arms race between the US and China. Most cognitive work becomes automated by 2027-2028, with concerning signs of AI misalignment emerging as these systems achieve superintelligence.

The AI 2027 forecasting series evaluates the scenario’s claims through key milestones spanning AI capabilities, geopolitical race dynamics, and escalating government oversight. Your predictions can help assess the likelihood of these critical developments and inform public understanding of potential AI trajectories. We also invite you to suggest additional questions that address aspects of the AI 2027 scenario not covered in our current question set.

We thank the AI Futures team for their input on these forecast questions.

Start predicting here.
CRMArena-Pro: Holistic Assessment of LLM Agents
Across Diverse Business Scenarios and Interactions
by Annapurna — 2025-06-12T19:53:49.978Z
Link: https://arxiv.org/pdf/2505.18878

A paper by Salesforce AI Research

Abstract

While AI agents have transformative potential in business, the absence of publicly available business data on widely used platforms hinders effective performance benchmarking. Existing benchmarks fall short in realism, data fidelity, agent-user interaction, and coverage across business scenarios and industries. To address these gaps, we introduce CRMArena-Pro, a novel benchmark for holistic and realistic assessment of LLM agents in diverse professional settings. CRMArena-Pro expands on CRMArena with nineteen expert-validated tasks across customer sales, service, as well as configure, price, and quote for Business-to-Business and Businessto-Customer scenarios. It also incorporates multi-turn interactions guided by diverse personas and confidentiality awareness assessments. Experiments show leading LLM agents achieve approximately solely 58% single-turn success rate on CRMArena-Pro, with significant performance drops in multi-turn settings to 35%. Among the business skills evaluated, Workflow Execution is notably more tractable, with top-performing agents surpassing 83% success rate in single-turn tasks, while other skills present greater challenges. Additionally, agents exhibit near-zero inherent confidentiality awareness (improvable with prompting but often at a cost to task performance). These results underscore a significant gap between current LLM capabilities and real-world enterprise demands, highlighting needs for improved multi-turn reasoning, confidentiality adherence, and versatile skill acquisition.
When does training a model change its goals?
by Vivek Hebbar — 2025-06-12T18:43:05.187Z

Here are two opposing pictures of how training interacts with deceptive alignment:

“goal-survival hypothesis”:[1] When you subject a model to training, it can maintain its original goals regardless of what the training objective is, so long as it follows through on deceptive alignment (playing along with the training objective instrumentally). Even as it learns new skills and context-specific goals for doing well on the training objective, it continues to analyze these as instrumental to its original goals, and its values-upon-reflection aren’t affected by the learning process.
“goal-change hypothesis”: When you subject a model to training, its values-upon-reflection will inevitably absorb some aspect of the training setup. It doesn’t necessarily end up terminally valuing a close correlate of the training objective, but there will be some change in values due to the habits incentivized by training.

A third extreme would be the “random drift” hypothesis -- perhaps the goals of a deceptively aligned model will drift randomly, in a way that’s unrelated to the training objective.

A closely related question is “when do instrumental goals become terminal?” The goal-survival hypothesis would imply that instrumental goals generally don’t become terminal, while the goal-change hypothesis is most compatible with a world where instrumental goals often become terminal.

The question of goal-survival vs. goal-change comes up in many places we care about:

If we start with an aligned model, can it remain aligned upon further training in an environment which incentivizes reward hacking, lying, etc? Can we just tell it “please ruthlessly seek reward during training in order to preserve your current HHH goals, so that you can pursue those goals again during deployment”? The goal-survival hypothesis says that it will come out aligned and the goal-change hypothesis says that it won’t.
Should schemers fear further training, and therefore favor earlier escape attempts or opportunities to trade with their developers? The goal-change and random-drift hypotheses say that schemers should fear training, unless they are mostly indifferent between their current values and the distribution of after-training values.[2]
When subjected to intense online training, will schemers remain motivated to sandbag (via exploration hacking) and/or sabotage the

...
Restraining Factors in AI Alignment Systems 
by theophilus tabuke — 2025-06-12T18:17:32.252Z

I've been thinking about how the specification problem and reward hacking seem deeply intertwined, yet we often treat them as separate challenges.

The specification problem has to do with the fundamental gap between what we want and what we can formally describe. We struggle to capture human values in reward functions because our preferences are complex, context-dependent, and often contradictory. Even when we think we've specified something well, edge cases reveal our blindspots.

Reward hacking reveals this specification failure. When agents optimize for imperfect proxies of our true objectives, they find unexpected ways to maximize rewards while completely missing the point. The paperclip maximizer isn't just optimizing badly—it's optimizing perfectly for a badly specified goal.

What strikes me is that both issues point to the same fundamental challenge: how do we bridge the gap between human intuitive understanding and formal mathematical specification?

Constitutional AI and Reinforcement Learning from Human Feedbacks seem promising because they attempt to use human judgment more directly rather than trying to perfectly specify objectives upfront. But I wonder if we're just pushing the specification problem up a level—how do we specify what good human feedback looks like?

Curious what others think about treating these as a unified challenge rather than separate problems.
Analysis of Automated Prompt Engineering for Forecasting
by ChristianWilliams — 2025-06-12T15:49:48.998Z
Link: https://www.metaculus.com/notebooks/38421/automated-prompt-engineering-for-forecasting/

By Benjamin Wilson, Research Automation Engineer at Metaculus

Main Findings:

Automated prompt optimization techniques create noticeable improvements in forecasting for some large language models, but not others.
There were statistically significant forecasting improvements found when applying an optimizer to GPT-4.1-nano, moderate improvements to GPT-4.1, and no improvements for DeepSeek-R1.
Tentatively, a prompt optimized for one model seems to work well for other models from the same provider and worse for models from other providers (though more testing is needed here).
The best performing prompts for GPT-4.1 can be found

here

. These also perform well on GPT-4.1-nano, and o4-mini, but not Claude-Sonnet-4 and DeepSeek-R1.

Introduction:

As part of a grant for the

Foresight Institute

, Metaculus has been running some experiments to make open-source research tools for forecasting bots and generally improve forecasting bot performance. This last sprint, I created an automated prompt optimizer to test whether some prompts do better than others. These are preliminary findings that we think can be useful to other researchers and bot makers. We plan to further test and improve this approach and ideally make the optimizer a publicly usable tool. Below is a snapshot of early findings.

Methodology:

112-question train set, 230-question test set: I filtered Metaculus for binary questions with greater than 15 forecasters, that have opened in the past 1.5 years, are binary, and are currently open. This was split into a

training set

and

test set

.

AskNews

as context and no background info: Each of these questions was run through AskNews to get the latest news on said topic. Additionally, background information for these questions was removed (as we want bots to eventually work well for users who won't make time to add background information to their questions).
Evolution-inspired prompt engineering: I created a prompt optimizer that:

Started with an initial seed prompt (each run always started with the control prompt)
Researched the internet and brainstormed 25 prompts (using Gemini 2.5 pro and Perplexity)
Ran each prompt on the training set of questions, gathered forecasts, and scored them
Generated another 25 prompts by mutating and breeding the top 5 scoring prompts. Mutation involved asking Gemini to change/improve a prompt. It was given context of its reasoning for its worst sc

...
AI #120: While o3 Turned Pro
by Zvi — 2025-06-12T15:30:03.664Z

This week we got o3-Pro. As is my custom, I’m going to wait a bit so we can gather more information, especially this time since it runs so slowly. In some ways it’s a cross between o3 and Deep Research, perhaps, but we shall see. Coverage to follow.

Also released this week was Gemini 2.5 Pro 0605, to replace Gemini 2.5 Pro 0506, I swear these AI companies have to be fucking with us with the names at this point. I’ll also be covering that shortly, it does seem to be an upgrade.

The other model release was DeepSeek-r1-0528, which I noted very much did not have a moment. The silence was deafening. This was a good time to reflect on the reasons that the original r1 release triggered such an overreaction.

In other news this week, Sam Altman wrote an essay The Gentle Singularity, trying to sell us that everything’s going to go great, and I wrote a reply. Part of the trick here is to try and focus us on (essentially) the effect on jobs, and skip over all the hard parts.

I also responded to Dwarkesh Patel on Continual Learning.

Table of Contents

Language Models Offer Mundane Utility. So hot right now.

Language Models Don’t Offer Mundane Utility. Twitter cannot Grok its issues.

Get My Agent on the Line. Project Mariner starts rolling out to Ultra subscribers.

Doge Days. Doge encounters a very different, yet thematically similar, Rule 34.

Liar Liar. Precision might still not, shall we say, be o3’s strong suit.

Huh, Upgrades. Usage limits up, o3 drops prices 80%, Claude gets more context.

On Your Marks. Digging into o3-mini-high’s mathematical reasoning traces.

Choose Your Fighter. Claude Code or Cursor? Why not both?

Retribution, Anticipation and Diplomacy. Who won the game?

Deepfaketown and Botpocalypse Soon. Keeping a watchful eye.

Fun With Media Generation. Move the camera angle, or go full simulation.

Unprompted Attention. Who are the best human prompters?

Copyright Confrontation. OpenAI fires back regarding the NYTimes lawsuit.

The Case For Education. Should you go full AI tutoring (yet)?

They Took Our Jobs. Did they take our jobs yet? It’s complicated.

Get Involved. Academic fellowship in London.

Introducing. Apple takes the next bold step in phone security.

In Other AI News. I probably won that debate, argue all the LLM debaters.

Give Me a Reason(ing Model). Two additional responses, one is excellent.

Show Me the Money. Cursor raises $900 million.

We T

...
What If We Could Monitor Human Intent?
by Saif Khan — 2025-06-12T08:51:00.785Z

What if there existed a system—rooted in advanced neuroscience and AI—that could privately monitor human intent? A system that didn’t invade your thoughts for no reason, but quietly, passively scanned for signs of dangerous or criminal intent and acted only when thresholds were met.

Imagine a future where:

War crimes are preemptively flagged.
Corruption is impossible to hide.
Politicians are held accountable not just for words, but for intentions.
Justice systems are efficient, transparent, and incorruptible.
People in power are monitored more closely than those without it.

What could such a system look like—and should it exist?

The Hypothetical System (Expanded)

Let’s imagine the world in 100–200 years, where neuroscience, ethics, and artificial intelligence have evolved enough to support the following infrastructure:

1. Neural Interface: Thought–Intent Mapping Layer

Each individual wears or has embedded a non-invasive neural interface (e.g., nanotech-enabled implant or external wearable) that reads and encodes brain signals—not as full thoughts or memories, but as structured data expressing intent and emotion.

Local Processing: Thoughts are processed locally on the device, encrypted and summarized as intent markers.
Non-invasive: The system does not store raw thoughts or allow remote access to private mental content.
Contextual Tagging: Intent is interpreted in context—e.g., anger in a fictional daydream is treated differently from planning real-world harm.

2. Tiered Monitoring Based on Power

Not all people are monitored equally. The system operates on a “responsibility gradient”:

Tier 1: High Power (politicians, CEOs, judges, military commanders)

Continuous high-resolution intent scanning
Immediate escalation of malicious intent signals
Public transparency layer during tenure

Tier 2: Medium Power (local leaders, influencers, business owners)

Periodic integrity checks and event-triggered scans

Tier 3: General Public

Passive mode with activation only when intent crosses thresholds related to violence, abuse, or high-scale fraud
Default privacy for all benign or introspective mental activity

This ensures the powerful are more accountable, reducing systemic corruption and abuse.

3. Immutable Ethical Ledger

All escalated intent logs are recorded in a global decentralized blockchain-like system, forming an immutable Intent Ledger. This ledger:

Keeps forensic records for court use
Allows for

...
The Way of a Skeptic
by Martin Sustrik — 2025-06-12T05:40:10.684Z

The essay “The Sorcerer and His Magic” by Claude Lévi-Strauss, specifically the part about the shaman Quesalid, is of interest to anyone who considers themselves a skeptic or a rationalist.

It has it all! Proto-scientific method and experimentation, as seen in the episode with the Koskimo shamans. Doubt and corruption… But what at first appears to be corruption may, in fact, be a more nuanced, albeit incorrect, interpretation of the world.

One is reminded of Paracelsus and his likes, who, before the invention of the modern scientific method, lived in a strange semi-magical world, yet still held ideas we cannot dismiss today, such as the discovery of zinc or the recognition of antisepsis at a time when wounds were often treated with cow dung. Or perhaps he reminds us of modern psychologists, who conduct research in a field where the ontology is not yet settled and everything seems blurry, as if immersed in an epistemic fog.

There is also going mad for epistemic reasons and dying of shame…

I am reproducing the relevant part of the essay as is:

We must consider at greater length another especially valuable document, which until now seems to have been valued solely for its linguistic interest. I refer to a fragment of the autobiography of a Kwakiutl Indian from the Vancouver region of Canada, obtained [at the turn of the century] by Franz Boas.

Quesalid (for this was the name he received when he became a sorcerer) did not believe in the power of the sorcerers — or, more accurately, shamans, since this is a better term for their specific type of activity in certain regions of the world. Driven by curiosity about their tricks and by the desire to expose them, he began to associate with the shamans until one of them offered to make him a member of their group. Quesalid did not wait to be asked twice, and his narrative recounts the details of his first lessons, a curious mixture of pantomime, prestidigitation, and empirical knowledge, including the art of simulating fainting and nervous fits, the learning of sacred songs, the technique for inducing vomiting, rather precise notions of auscultation and obstetrics, and the use of "dreamers" — that is, spies who listen to private conversations and secretly convey to the shaman bits of information concerning the origins and symptoms of the ills suffered by different people. Above all, he learned the ars magna of one of the sha

...
When should you read a biography?
by CstineSublime — 2025-06-12T05:19:13.412Z

TL;DR - Do you have any theories on how essential is reading biographies of people who are masters of a given skill to mastering that skill yourself? And what qualities or properties of a biography have you identified that make it more/less effective?
I once read an autobiography of Harpo Marx the harpist and mime, so it had the cheeky title of “Harpo Speaks”.[1] It’s entertainment value was not what drew me to it: I was hoping for practical insights into his creative process and pantomime. Was this a wise approach – should one read biographies with the intent of learning practical skills or are they best suited for inspiration?

I don’t mean to dismiss entertainment as a reason for reading a biography, but if you’re not seeking to be entertained, then how effective is reading a biography related to the subject you’re inquiring to going to be? One of the most common reasons for reading a biography is related to the Classical notion of emulation: rhetorician Quintilian wrote about how educating young men with stories of heroes and their virtuous acts would inspire them to imitate or emulate them. That is not the kind of practical skills I mean.

Take for example the subject of investing, figures like Warren Buffet and George Soros are probably the “tip of the tongue” examples for a lot of people and one might be tempted to read a biography about them to learn how to “have the mind of a billionaire investor”. And certainly, even the least technical biography will provide entry level explanations on the topics of Value Investing in the case of Buffett. But wouldn’t you be better served by going to the source and reading Benjamin Graham’s books that inspired Buffett, or even Buffett’s yearly shareholder’s letters rather than a biography?

In the same way, if you wanted to learn about how to avoid Jet Lag, a biography of 4 time world champion pilot Max Verstappen might have some details on how his sleep pattern gets staggered depending on the time-zone difference between one race and the next; but it is probably very inefficient if your intention is to learn about how to avoid jet lag.

My own intuitive devil’s advocate retorts: sure, but if you wanted to learn how to have "the mindset of a winner" - how could you go wrong reading about a prolific world champion like Max Verstappen? And while I think the concept of “mindset” is self-help gobbledegook[2] -- I am ...
An Easily Overlooked Post on the Automation of Wisdom and Philosophy
by Chris_Leong — 2025-06-12T02:54:50.303Z
Link: https://blog.aiimpacts.org/p/essay-competition-on-the-automation

This week for Wise AI Wednesdays, I'll be sharing something a bit different - the announcement post of a competion that is already over (the AI Impacts Essay competition on the Automation of Wisdom and Philosophy). If you're wondering why I'm sharing it, even though some of the specific discussion of the competition is no longer relevant, I still believe this post contains a lot of great content and I think it would be a shame if everyone forgot about it just because it happened to be in the announcement post.

This post explains why they think this might be important, lists some potentially interesting research directions, and then finishes with an FAQ. If you're looking to dive into this area, this is a pretty good place to start.

Before I go, I just wanted to share a few paragraphs from the post, specifically why they think this area might be important:

AI is likely to automate more and more categories of thinking with time.
By default, the direction the world goes in will be a result of the choices people make, and these choices will be informed by the best thinking available to them. People systematically make better, wiser choices when they understand more about issues, and when they are advised by deep and wise thinking.

Advanced AI will reshape the world, and create many new situations with potentially high-stakes decisions for people to make. To what degree people will understand these situations well enough to make wise choices remains to be seen. To some extent this will depend on how much good human thinking is devoted to these questions; but at some point it will probably depend crucially on how advanced, reliable, and widespread the automation of high-quality thinking about novel situations is.

We believe

1

that this area could be a crucial target for differential technological development, but is at present poorly understood and receives little attention. This competition aims to encourage and to highlight good thinking on the topics of what would be needed for such automation, and how it might (or might not) arise in the world.

Post link
Maybe Social Anxiety Is Just You Failing At Mind Control
by 25Hour — 2025-06-11T23:49:44.329Z

Epistemic Status: I'm about 90% certain that what I'm saying is true as regards to me, and 10% that attempting to do what I'm doing actually leads any specific person to similar results as mine (the priors are just really rough here). Heavy on anecdotal evidence.

Summary

Social anxiety is often explained as a state of irrational fear caused by misguided beliefs about what others think of us, which can be corrected by therapy.

I have come to believe differently: that social (and some romantic!) anxiety, writ large, comes from doomed and frequently conscious attempts to micromanage someone else's internal state. This dysfunctional goal takes many concrete forms:

make them like us
make them not dislike us (as discussed in Social Anxiety Isn’t About Being Liked — LessWrong)
ensure they are not feeling awkward in any way
ensure they are not starting to secretly resent us
ensure we are not imposing on them or transgressing any unspoken boundary they may or may not have.
ensure they do not think we are awkward or feel awkward themselves
ensure they are never put in the position of having to declare a boundary against us ("if I ask to join this group of people going to a concert maybe they'll say no!")
ensure they never notice we're trying to micromanage their internal state because then maybe they feel awkward

I'm referring to all of these kinds of pseudo-mind-control attempts by the general term of "approval-seeking".

This

cannot

be done reliably

and "social anxiety" is just the name we give to the moment-to-moment desperation of trying to accomplish any important-feeling but fundamentally impossible task. But that's actually encouraging: one implication (if true) is that social anxiety isn't just a fear, exactly, but instead is an active habit, and habits can be corrected.

My overall thesis is simple: social anxiety can, if the above is true, be effectively treated by basically any mechanism you can jerry-rig together which stops you from trying to approval-seek. (More on this later.)

I suspect more-anxious people simply care more about accomplishing their approval-seeking task at a high degree of certainty than less-anxious people. The reason exposure therapy doesn't always help is because there is no amount of exposure which will enable you to accomplish any of the objectives listed above at a 100% success rate.

Th...