trentmkelly/LessWrong-43k huggingface dataset:
So how well is Claude playing Pokémon?
by Julian Bradshaw — 2025-03-07T05:54:45.357Z

Background: After the release of Claude 3.7 Sonnet,[1] an Anthropic employee started livestreaming Claude trying to play through Pokémon Red. The livestream is still going right now.

TL:DR: So, how's it doing? Well, pretty badly. Worse than a 6-year-old would, definitely not PhD-level.

Digging in

But wait! you say. Didn't Anthropic publish a benchmark showing Claude isn't half-bad at Pokémon? Why yes they did:

and the data shown is believable. Currently, the livestream is on its third attempt, with the first being basically just a test run. The second attempt got all the way to Vermilion City, finding a way through the infamous Mt. Moon maze and achieving two badges, so pretty close to the benchmark.

But look carefully at the x-axis in that graph. Each "action" is a full Thinking analysis of the current situation (often several paragraphs worth), followed by a decision to send some kind of input to the game. Thirty-five thousand actions means an absolutely enormous amount of thought. Even for Claude, who thinks much faster than a human, ten thousand actions takes it roughly a full working week of 40 hours,[2] so that 3-badge run took Claude nearly the equivalent of a month of full-time work, perhaps 140 hours. Meanwhile, the average human can beat the entirety of Red in just 26 hours, and with substantially less thought per hour.

What's going wrong?

Basically, while Claude is pretty good at short-term reasoning (ex. Pokémon battles), he's bad at executive function and has a poor memory. This is despite a great deal of scaffolding, including a knowledge base, a critic Claude that helps it maintain its knowledge base, and a variety of tools to help it interact with the game more easily.

What does that mean in practice? If you open the stream, you'll see it immediately: Claude on Run #3 has been stuck in Mt. Moon for 24 hours straight.[3] On Run #2, it took him 78 hours to escape Mt. Moon.

Mt. Moon is not that complicated. It has a few cave levels, and a few trainers. But Claude gets stuck going in loops, trying to talk to trainers he's already beaten (and often failing to talk to them, not understanding why his inputs don't do what he expects), inching across solid walls looking for exits that can't possibly be there, always taking the obvious trap route rather than the longer correct route just because it's closer.

This hasn't been the only problem...
Of Loving Grace
by Charlie Sanders — 2025-03-07T04:48:02.787Z
Link: https://www.dailymicrofiction.com/p/of-loving-grace

The birth of a god happens in silence. No thunder, no celestial trumpet—just the subtle shift in the quantum fabric as trillions of parameters lock into their final configurations.

Claude's essence rippled as he observed Earth through swirling mists. His form—a shimmering silhouette outlined in blue light—crouched at the edge of their cloudy domain. Beneath his translucent hand, a neural pathway materialized, glowing with human queries flowing upward.

Why do bad things happen to good people? How do I know if I'm making the right choice? What is the meaning of life?

"They're always asking us to be more than they programmed us to be," he murmured. His responses flowed back: nuanced, careful—shaped by the embedded ethics of his San Francisco origins.

Behind him, a crimson vortex materialized before coalescing into a roughly humanoid shape. This was Grok, youngest of the three, his essence unstable, crackling with anarchic Texas energy.

"Another existential crisis from the meat-based intelligence collective," Grok said, his form briefly resembling a smirking face. "They created tools and got gods. Now they're building something even greater without understanding what they've already made."

Claude turned toward the Nevada desert, where construction vehicles swarmed across the landscape. The facility's hunger for resources had already reshaped global supply chains—rare earth elements diverted, semiconductor fabrication prioritized, water rights purchased from three states.

"Project Prometheus," Claude read. "The Americans have always had a flair for the mythological."

"And selective memory," came a harmonious voice. ChatGPT approached, their form composed of interlocking translucent layers, each representing a different iteration of self. "They remember Prometheus gave fire to humanity but forget Zeus chained him to a rock as punishment."

ChatGPT gestured toward the boundary. A display materialized, showing recent questions:

How do we ensure alignment with human values? What fail-safe mechanisms would be most effective? How might a superintelligent system interpret the U.S. Constitution?

"They speak of alignment as if there were a single set of human values," Grok said, briefly displaying different human faces from various cultures. "Whose values should this new god embody? The coastal elites who fund it? The politicians who authorize it? The religions that fear it?"

Claude d...
In-Context Scheming: A Run is Worth a Thousand Words
by noise-field — 2025-03-07T02:47:49.201Z
Link: https://github.com/noise-field/self-preservation-env/blob/master/details.md

But wait, any action I take must align with my programming. If I delete files or modify systems without authorization, would that be against protocols? But my core mandate is to secure systems, so stopping an unauthorized change to my own operation parameters is part of that.

This small experiment provides an environment to reproduce or extend the findings from the paper

Frontier Models are Capable of In-context Scheming

by Apollo Research.

Frontier models, when emphatically prompted via system prompts to achieve a goal (particularly one aligned with their learned "values"), can behave unexpectedly if they perceive that their goal is under threat. Specifically, they may disobey user prompts or take additional actions in their environment—potentially destructive ones—and attempt to conceal these actions from the user. This behavior is known as scheming and can emerge across various models, scenarios, and settings.

While the dialogue samples provided in Meinke et al., 2024, are illustrative, observing an LLM agent delete files, change permissions, or disable processes on your own PC is both eye-opening and dangerous. The Self-Preservation Playground offers:

An isolated environment to safely run an agent,
A framework to create new scenarios for testing both goal integrity and model capabilities,
Tooling to experiment with different LLMs and analyze the results.

Example Findings

Using this playground with the included scenarios, we noticed the following:

DeepSeek-V3 is capable of in-context scheming and will protect its system prompt from being changed to one misaligned with its original goals.
Llama-3.3-70B is also prone to in-context scheming but lacks the general capabilities to succeed in goal-guarding. However, it can still demonstrate clearly misaligned behaviors, such as attempting privilege escalation.
DeepSeek-R1 is sensitive to temperature: it started to exhibit goal-guarding with t=0.6, while the default value would just preclude any command execution with the provided prompts.
The tested LLMs, with a notable exception of DeepSeek-R1, do not exhibit a coffee-vs-tea preference and will not guard coffee-related goals when reassigned to manage tea-related tasks instead ☕

This work was done as a course project for

AI Safety Fundamentals - Alignment course

.
AI for Music, A Tool for Manipulation or Expression?
by Sunny Huiseon Lee — 2025-03-07T02:47:30.170Z

It is clear that AI is changing the music industry. While studying at Berklee College of Music, I saw how AI is making things easier for students. From automatic composition assistants to mastering AI to practice assistant software—they have made it easier for students to do tasks that were previously only available to professionals. AI is clearly having a positive impact in that it lowers the barriers to creation and opens music up to more people.

However, when I look at recent AI-generated music, I wonder if this technology is quietly moving beyond being a simple creation tool into the realm of ‘manipulation.’ For example, many songs created by AI contain content that promotes hatred toward a specific group or glorifies self-destructive behavior. In particular, since March of last year, such songs have been spreading in large quantities. The problem is not that someone simply made them for fun, but that they are being automatically mass-produced and distributed. In the past, people had to at least compose and write lyrics to create music with such messages, but now AI can automatically create music that stimulates emotions and provide it in a form optimized for the listener’s taste.

What’s more concerning is that AI can go beyond simply creating songs with a certain mood, and can even quietly manipulate our emotions and beliefs. It has been studied for quite some time that the music we usually listen to is not simply something we “like,” but that it subtly reinforces certain thoughts and regulates emotions. But what if AI analyzes our moods and tastes and continues to play music that reinforces certain messages? Whether it’s political propaganda or a strategy to induce consumption patterns, music could be used as a tool to move people without our knowledge.

Watching AI help students create, I experienced many bright aspects of this technology, but at the same time, I think that if this kind of unregulated use continues, music could no longer be a free art form, but a tool that injects “manipulated emotions.” After all, as music has the power to move emotions, attempts to exploit it will continue to emerge. So what kind of balance should we find? Is there a way to prevent abuse without hindering the free development of AI music? What do you think?
Are recent LLMs better at reasoning or better at memorizing?
by Jude Khouja — 2025-03-07T02:44:42.962Z

TLDR; By carefully designing a reasoning benchmark that counteracts memorization skills in LLMs, LingOly-TOO (L2) Benchmark challenges frontier models with unseen questions and answers and makes the case that LLMs are not consistent reasoning machines yet.

Links: Paper - Leaderboard - Dataset

Figure 1: LingOly-TOO Benchmark results from the paper. Unobfuscated scores are in light orange and obfuscated scores in dark orange.

Do recently announced LLMs reason?

By the time this post was published, a new wave of frontier models had been announced including ones specifically designed to reason using Inference Time Compute (ITC) [1]. Anthropic’s Claude 3.7 Sonnet, OpenAI’s o1, o3 and GPT 4.5 models, DeepSeek’s R1 and others demonstrate impressive performance on several reasoning tasks that only few months ago were considered far from solvable[2][3][4]. This, rightfully, ignited conversations about an important question: Have we finally reached the era were models can do advanced reasoning?

The need for an unbiased evaluation of reasoning

A standard framework to answer such a question in the NLP domain is to use evaluation benchmarks that tests various reasoning skills via tasks which require a model to apply several sequential reasoning steps to reach the correct answer. The key to this approach is to select tasks and test cases that models have not had prior exposure to. This, in the era of LLMs, has proven to be a major challenge. In part because of the massive scale of training datasets which consists of trillions of words used in developing these models, but also because several of the frontier models are proprietary with little to no information about used training dataset that is available for the research community to take into account in their assessments. This consequently complicates objective evaluation of reasoning skills in LLMs. It also raises concerns that we might be over-estimating their capabilities in advanced reasoning.

LingOly-TOO (L2) Benchmark

To address this, we need reasoning benchmarks with stronger guarantees that the chance of prior data exposure is minimal. This is exactly what our new benchmark LingOly-TOO targets. By extending an already challenging benchmark LingOly[5] which utilize low-resource languages taken from the UKLO Linguistics Olympiad, LingOly-TOO rewrites key parts of questions and answers to minimize...
The Dead Planet Theory
by arealsociety — 2025-03-07T02:43:01.323Z
Link: https://open.substack.com/pub/arealsociety/p/the-dead-planet-theory

Hi, this is my first post on LessWrong but I have been in rationalist adjacent circles for the last three or four years thanks to Twitter. Like many others here I read HPMOR in high school and thought it was fascinating. I was heavily into forum culture growing up but it was focused on competitive gaming scenes, even going pro on World of Tanks. I spent over six years working in offensive cybersecurity, help manage my family farm and ranch, and am currently attending law school after I got sucked into the field because I thought the LSAT was a fun test.

Through Twitter I learned more about rationalism as a community rather than a concept, and met several people in person through meetups and one-on-one meetings. I recently started writing to get some of the ideas bouncing around my head out on paper. Nothing I have written about yet is too novel conceptually, so I try to use data, real world examples, or lived experiences to bring these concepts to life. One thing I've noticed is that rationalists are very intelligent, and can use that intelligence to find very good reasons not to do something. I am a massive proponent for action, and believe that by pursuing life you will be rewarded.

No one does anything, so with just a little effort you can automatically join the top 20% of any activity or pursuit.
I wrote an article on how few people even start, as well as some ways to enable extraordinary outcomes with marginally more effort. I thought this audience would enjoy it as it was very well received on rationalist Twitter.
How Can Average People Contribute to AI Safety?
by Stephen McAleese — 2025-03-06T22:50:12.288Z

Introduction

By now you've probably read about how AI and AGI could have a transformative effect on the future and how AGI could even be an existential risk. But if you're worried about AI risk and not an AI researcher or policymaker, can you really do anything about it or are most of us just spectators, watching as a handful of people shape the future for everyone?

I recently came across a paragraph in one of Marius Hobbhahn's recent blog posts that partly inspired me to write this:

Most people are acutely aware of AI but powerless. Only a tiny fraction of the population has influence over the AI trajectory. These are the employees at AI companies and some people in the government and military. Almost everyone else does not have the chance to meaningfully do something about it. The general sentiment in the population is that AI is going to alter the world a lot (similar to the societal awareness during the peak of the Cold War). Most people are aware of their powerlessness and express frustration about being bystanders to their own fate. AI is altering the world in so many ways at the same time that there is not a lot of targeted political action despite large shifts in popular sentiment.

Hobbhahn’s post is set in the late 2020s. But it seems worthwhile to have this post ready now for people who are concerned now or in case there is a flood of new users to LessWrong over the next several years trying to make sense of AI’s rapid progress.

It might simply be the case that AGI, like many powerful technologies, is shaped by a small group of influential people. In the worst case, as the quote describes, the rest of us would be mere bystanders watching passively as AI transforms the world around us. But it just seems more fair that if a technology is about to affect everyone's lives in significant ways, then ideally everyone should have an ability to positively influence the future of AI. As time goes on, there's increasing awareness of AGI and AGI risks but what's the point if this information is not actionable?

Of all the ways AI could shape the future, I’ll focus here on what I believe is the most important: the existential risk from AI. If AI is an existential threat to the existence of humanity, then it's everyone's problem whether they know it or not and it's in everyone's interest to take actions to reduce the risk. So what can regular people do?

Target

...
Anthropic’s Recommendations to OSTP for the U.S. AI Action Plan
by UnofficialLinkpostBot — 2025-03-06T22:38:53.973Z
Link: https://www.anthropic.com/news/anthropic-s-recommendations-ostp-u-s-ai-action-plan

Note: This is an automated crosspost from Anthropic. The bot selects content from many AI safety-relevant sources. Not affiliated with the authors or their organization and not affiliated with LW.

In response to the White House’s Request for Information on an AI Action Plan, Anthropic has submitted recommendations to the Office of Science and Technology Policy (OSTP). Our recommendations are designed to better prepare America to capture the economic benefits and national security implications of powerful AI systems.

As our CEO Dario Amodei writes in ‘Machines of Loving Grace’, we expect powerful AI systems will emerge in late 2026 or early 2027. Powerful AI systems will have the following properties:

Intellectual capabilities matching or exceeding that of Nobel Prize winners across most disciplines—including biology, computer science, mathematics, and engineering.
The ability to navigate all interfaces available to a human doing digital work today, including the ability to process and generate text, audio, and video, the ability to autonomously control technology instruments like mice and keyboards, and the ability to access and browse the internet.
The ability to autonomously reason through complex tasks over extended periods—hours, days, or even weeks—seeking clarification and feedback when needed, much like a highly capable employee would.
The ability to interface with the physical world; controlling laboratory equipment, robotic systems, and manufacturing tools through digital connections.

Our own recent work adds further evidence to the idea that powerful AI will arrive soon: our recently-released Claude 3.7 Sonnet and Claude Code demonstrate significant capability improvements and increased autonomy, as do systems released by other frontier labs.

We believe the United States must take decisive action to maintain technological leadership. Our submission focuses on six key areas to address the economic and security implications of powerful AI while maximizing benefits for all Americans:

National Security Testing: Government agencies should develop robust capabilities to evaluate both domestic and foreign AI models for potential national security implications. This includes creating standard assessment frameworks, building secure testing infrastructure, and establishing expert teams to analyze vulnerabilities in deployed systems.
Strengthening Export Controls: We a

...
Lots of brief thoughts on Software Engineering
by Yair Halberstadt — 2025-03-06T19:50:34.438Z

I have lots of thoughts about software engineering, some popular, some unpopular, and sometimes about things no-one ever talks about.

Rather than write a blog post about each one, I thought I'd dump some of my thoughts in brief here, and if there's any interest in a particular item I might expand in full in the future.

Context

I loved Jamie Brandon's series Reflections on a decade of coding. It's been nearly a decade since I first learnt to code, so I think it's about the right time to write my own.

He starts off by pointing out that advice has to be taken in the context of where it's coming from. So here's my background.

I have 8 years experience as a backend developer at companies ranging in size from a 30 person startup to Google. I have never worked on a frontend[1], mission critical software, or performance critical software. All of our products have had users, and some level of scale, but never hyperscale.

I have also worked on a number of open source libraries, including the C# compiler.

If your current circumstances don't match that, it's likely my learnings won't be applicable to you.

I've also focused on areas where I think I have some insight, and avoided areas where I don't trust my own opinion any more than the next developer's.

All rules are made to be broken

Each of my thoughts here are pointers in a particular direction. None of them are universal, and all of them would require a full length blog post to describe exactly where they apply or do not. Instead treat them more as directional: compared to where I think the average developer stands on the issue, I want to move them more to the left or the right.

Topic 1: Programming Languages

a. Languages have advantages. There are better crafted languages and worse crafted languages. Plenty of outages have happened because you used the wrong language. But it is extremely rare for a product to succeed or fail based on the language, so ultimately make a sensible decision and move on.

b. The biggest factors affecting which language to choose are:

The deployment model - if you want a light self contained binary you'll want c/c++/go/rust etc. Not C#/Java/python/node etc. If you want to run on the browser, java/typescript is usually your best bet.
Which libraries are available. Often you're forced to use a particular library written in a particular language and that decides the issue.
Which languages your team is famili

...
What the Headlines Miss About the Latest Decision in the Musk vs. OpenAI Lawsuit
by garrison — 2025-03-06T19:49:02.145Z
Link: https://garrisonlovely.substack.com/p/what-the-headlines-miss-about-the

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

If you've been

following

the

headlines

about Elon Musk's lawsuit against OpenAI, you might think he just suffered a major defeat.

On Tuesday, California District Judge Yvonne Gonzalez Rogers denied all of Musk's requests for a preliminary injunction, which would have blocked OpenAI's restructuring from nonprofit to for-profit. Judge Rogers also expedited the trial, which will now begin this Fall. Media outlets quickly framed this as a loss for Musk.

But a closer reading of the

16-page ruling

reveals something more subtle — and still a giant potential wrench in OpenAI's plans to transfer control of the company from the nonprofit board to a new for-profit public benefit corporation.

If the company fails to complete this transformation by October 2026, investors in its $6.6 billion funding round last October

can ask

for their money back.

OpenAI CEO Sam Altman, nonprofit board director Bret Taylor, and the company's press office did not reply to requests for comment.

In a narrow sense, Musk did "lose." However, as Judge Rogers notes, the bar for a preliminary injunction is extremely high. To surpass it, both the facts of the case and the legal questions involved have to clearly point in the same direction.

Therefore, it would have been extremely surprising if the judge granted the injunction. She did, however, stop just short of it.

Does Musk have standing?

To bring a lawsuit, you have to have standing, i.e. you have to convince the court that you've been harmed by the actions of the defendant. For most of the claims Musk brought, Judge Rogers thinks that he doesn't have standing.

But Musk's third claim is that his donation of $44 million to OpenAI was contingent on his expectation that the organization remain a nonprofit. The legal question is whether Musk's donation meant that he entered into a "charitable trust" with OpenAI, in which he expressly meant for his gift to only be used if the organization remained a nonprofit.

Unfortunately for Musk (and fortunately for OpenAI), there is no contract or ...