trentmkelly/LessWrong-43k huggingface dataset:
Legal Supervision of Frontier AI Labs is the answer.
by Gauraventh — 2025-05-05T13:36:38.131Z
Link: https://robertandgaurav.substack.com/p/legal-supervision-of-frontier-ai

If the biggest threat model from AI systems comes from internal deployment, then the correct governance move is to establish independent legal supervisors for frontier AI labs[1].

Steven Adler recently argued against relying on a "race to the top," where frontier labs compete to be the safest when deploying models.

‘A race to the top can improve AI safety, but it doesn’t solve the ‘adoption problem’: getting all relevant developers to adopt safe enough practices.’

We should be sceptical of actors racing to build what could become the most powerful technology in human history and then saying they'll compete to be safe. There’s already plenty of evidence for this. Google didn’t release model evaluations for Gemini 2.5 until after they were publicly criticised. It is unclear whether they ever intended to do so. OpenAI, meanwhile, is attempting to convert its non-profit structure and has made several other choices that undermine safety.

If a serious race to the top were truly underway, we might as well go home.

But we cannot go home.

If these labs can’t be trusted, then someone else must keep them in check. So where are the checks and balances for AI labs?

Internal structures like boards or long-term governance trusts do have some power to provide oversight. But they are too far removed to catch problems early, and too reliant on information the lab chooses to share.

What is needed is a higher standard of scrutiny: deeper inspection, independent testing, and people with direct access to the models and their behaviour.

This level of oversight requires more than internal governance. It calls for an independent body with a clear legal mandate. Not friendly auditors chosen by the labs, but supervisors with the authority to inspect systems, test for risks like scheming, and intervene before deployment. They must have the power to file warrants, demand documentation, and compel access when necessary. Legislation should grant these powers and set clear boundaries for how they are used.

There is good reason to be sceptical of applying existing legal frameworks to frontier AI. Taking something off the shelf, like liability or contract law, probably will not work by default. And we should also be sceptical that the law can keep pace with how quickly this technology is moving. 'Statutory law and case law never keep up with Moore’s law.'

But that is not a reason to give up on legal ap...
The crucible — how I think about the situation with AI
by owencb — 2025-05-05T13:18:28.429Z
Link: https://strangecities.substack.com/p/the-crucible

The basic situation

The world is wild and terrible and wonderful and rushing forwards so so fast.

Modern economies are tremendous things, allowing crazy amounts of coordination. People have got really very good at producing stuff. Long-term trends are towards more affluence, and less violence.

The enlightenment was pretty fantastic not just for bringing us better tech, but also more truthseeking, better values, etc.

People, on the whole, are basically good — they want good things for others, and they want to be liked, and they want the truth to come out. This is some mix of innate and socially conditioned. (It isn’t universal.) But they also often are put in a tight spot and end up looking out for themselves or those they love. The hierarchy of needs bites. Effective altruism often grows from a measure of privilege.

The world is shaped by economics and by incentives and by institutions and by narratives and by societal values and by moral leadership and by technology. All of these have a complex interplay.

AI enters the picture

“AI” is a cluster of powerful technologies which are likely to reshape the world. Each of economics and incentives and institutions and narratives and societal values and moral leadership will (I expect) be profoundly impacted by advanced AI. And, of course, AI will be used to shape advanced technologies themselves.

From a zoomed-out perspective, there are three impacts of this AI transition which matter most[1]:

Technological progress will accelerate — automation of research means the world will get even faster and crazier
New kinds of entities could be making the choices that matter — either purely artificial agents, or new hybrid institutions which incorporate AI deeply in their decision-making
It will become far easier to centralize control — a single entity (AI system, or organization, or person) could end up with robust and enduring power over a large domain, or even the entire world

This is … kind of wild. 2) and 3) could lead to profound changes to the way the world works, and 1) means this entire thing might happen very quickly (and that people are therefore more likely to fumble it).

From the perspective of most people in rich countries today, this is all pretty disturbing. We have enjoyed a good measure of affluence and of stability. This reshaping of the world will bring further affluence, but it is vast and unknown and could easily le...
Proposal: Liquid Prediction Markets for AI Forecasting
by Jesse Richardson — 2025-05-05T05:13:48.965Z

Background

Polymarket is a prediction market platform where people can trade contracts representing the probability of events occurring (e.g., "Who will win the 2024 Presidential election").
Trading volume on Polymarket is heavily influenced by liquidity rewards, a program which pays traders to place competitive bids and offers in various markets, thereby increasing liquidity and encouraging more trades. Polymarket currently spends an average of $17K/day on liquidity rewards, and can range from $1/day to as much as $5K/day on various individuals.
Polymarket pays for the liquidity rewards itself (thanks, VC money!), and chooses how high to set the rewards on each market.

The problem

Despite significant interest in AI forecasting, there is a lack of liquid high-quality prediction markets on the future of AI.
Polymarket has ~25 markets on AI, but most of them aren't very interesting (e.g. predicting the Chatbot Arena leaderboard) and they all have pretty low liquidity, which is downstream of low liquidity rewards. For example, I'm pretty interested in "Will AI win an IMO gold medal in 2025?", but the rewards are set at a relatively low $5/day and the order book is pretty thin as a result.
Other prediction markets sites such as Kalshi suffer from similar pitfalls.

The proposal

I've been speaking a bit with a member of the Polymarket team and they are interested in having someone fund higher liquidity rewards on AI markets -- these could be existing markets or new markets that are specifically requested.
There's not yet a formal process for this to happen, and it's not guaranteed Polymarket will be able to make it work, but it seems pretty likely.
As a result, I am looking to partner with individuals or organizations who want to help make high-liquidity prediction markets a reality for AI forecasting.
I am primarily interested in hearing from potential funders, but also from people in AI forecasting who want to come up with new markets that should receive liquidity rewards funding, and help write rules for those markets.
To begin with, I'm hoping to raise $55K, which could fund ten markets for six months with liquidity rewards of $30/day.

Will this result in better AI forecasts?

Boosting liquidity rewards on AI prediction markets will definitely increase the trading volume on these markets, of that there is no doubt. The question is a) how much, and b) will that translate to mo

...
Why “Solving Alignment” Is Likely a Category Mistake
by Nate Sharpe — 2025-05-05T04:26:50.825Z

A common framing of the AI alignment problem is that it's a technical hurdle to be overcome. A clever team at DeepMind or Anthropic would publish a paper titled "Alignment is All You Need," everyone would implement it, and we'd all live happily ever after in harmonious coexistence with our artificial friends.

I suspect this perspective constitutes a category mistake on multiple levels. Firstly, it presupposes that the aims, drives, and objectives of both the artificial general intelligence and what we aim to align it with can be simplified into a distinct and finite set of elements, a simplification I believe is unrealistic. Secondly, it treats both the AGI and the alignment target as if they were static systems. This is akin to expecting a single paper titled "The Solution to Geopolitical Stability" or "How to Achieve Permanent Marital Bliss." These are not problems that are solved; they are conditions that are managed, maintained, and negotiated on an ongoing basis.

The Problem of "Aligned To Whom?"

The phrase "AI alignment" is often used as shorthand for "AI that does what we want." But "we" is not a monolithic entity. Consider the potential candidates for the entity or values an AGI should be aligned with:

Individual Users: Aligning to individual user preferences, however harmful or conflicting? This seems like a recipe for chaos or enabling malicious actors. We just experienced an example of how this can go wrong with the GPT-4o update and subsequent rollback, wherein positive user feedback resulted in an overly sycophantic model personality (that was still rated quite highly by many users!) with some serious negative consequences.
Corporate/Shareholder Interests: Optimizes for proxy goals (e.g., profit, engagement) which predictably generate negative externalities. Subject to Goodhart's Law on a massive scale.
Democratic Consensus: Aligning to the will of the majority? Historical precedent suggests this can lead to the oppression of minorities. Furthermore, democratic processes are slow, easily manipulated, and often struggle with complex, long-term issues.
AI Developer Values: Aligning to the personal values of a small, unrepresentative group of engineers and researchers? This introduces the biases and blind spots of that specific group as the de facto global operating principles. We saw how this can go wrong with the Twitter Files - imagine if that was inst

...
AI, Animals, & Digital Minds 2025: apply to speak by Wednesday!
by Alistair Stewart — 2025-05-05T00:56:10.981Z

AI, Animals, & Digital Minds (AIADM) 2025 is a one-day conference and two-day unconference exploring the intersection of AI and sentient nonhumans, both biological (i.e. animals) and potentially artificial.

Learn more and apply here

Taking place in London and virtually from Friday 30th May until Sunday 1st June – the weekend before EAG London – the event will bring together thought leaders with backgrounds in philosophy, animal welfare science, industry, AI ethics and safety, and government and policy.

On Friday there will be a livestreamed conference for up to 130 attendees at University College London, where speakers will include David Pearce, Jeff Sebo and Joey Savoie. At the weekend there will be a participant-led unconference for up to 60 participants at the Ambitious Impact office in Spitalfields, with a separate virtual unconference between 4 and 7pm UTC on Saturday and Sunday.

We are extending two deadlines until 11.59pm BST, Wednesday 7th May:

Call for speakers
Early bird discount on conference tickets (use promotion code EB25AIADM)

A few things to note:

We don't have many speaker slots left for the conference (on Friday).
There are slots open for talks, discussion, workshops and other sessions during the unconference (at the weekend). You don't have to apply to speak now to deliver a lightning talk or lead a discussion group during the unconference – if you're accepted to the unconference you will be able to propose a talk or session during the event.
Throughout AIADM we are particularly interested in talks and discussions which have a focus on delivering concrete policy-related outcomes.
We aren't expecting to sell out of Friday conference tickets before the middle of May.
We are rapidly approaching capacity for the in-person weekend unconference due to demand and space constraints. You will increase your chances of being accepted to the unconference by applying in the next few days, and by being specific about the questions you're interested in discussing in your application.

More information and all relevant links can be found at www.aiforanimals.org.

Feel free to comment or email londonconference@aiforanimals.org if you have any questions or thoughts!
Notes on the Long Tasks METR paper, from a HCAST task contributor
by abstractapplic — 2025-05-04T23:17:50.851Z

I contributed one (1) task to HCAST, which was used in METR’s Long Tasks paper. This gave me some thoughts I feel moved to share.

ETA: I've made some substantial changes thanks to responses by the original authors.

Regarding Baselines and Estimates

METR’s tasks have two sources for how long they take humans: most of those used in the paper were Baselined using playtesters under persistent scrutiny, and some were Estimated by METR.

I don’t quite trust the Baselines. Baseliners were allowed/incentivized to drop tasks they weren’t making progress with, and were – mostly, effectively, there’s some nuance here I’m ignoring – cut off at the eight-hour mark; Baseline times were found by averaging time taken for successful runs; this suggests Baseline estimates will be biased to be at least slightly too low, especially for more difficult tasks.[1]

I really, really don’t trust the Estimates[2]. My task was never successfully Baselined, so METR’s main source for their Estimate of how long it would take – aside from the lower bound from it never being successfully Baselined – is the number of hours my playtester reported. I was required to recruit and manage my own playtester, and we both got paid more the higher that number was: I know I was completely honest, and I have a very high degree of trust in the integrity of my playtester, but I remain disquieted by the financial incentive for contractors and subcontractors to exaggerate or lie.

When I reconstructed METR’s methodology and reproduced their headline results, I tried filtering for only Baselined tasks to see how that changed things. My answer . . .

. . . is that it almost entirely didn’t. Whether you keep or exclude the Estimated tasks, the log-linear regression still points at AIs doing month-long tasks in before 2030 (if you look at the overall trend) or 2028 (if you only consider models since GPT-4o). My tentative explanation for this surprising lack of effect is that A) METR were consistently very good at adjusting away bias in their Estimates and/or B) most of the Estimated tasks were really difficult ones where AIs never won, so errors here had negligible effect on the shapes of logistic regression curves[3].

Regarding Task Privacy

HCAST tasks have four levels of Task Privacy:

“fully_private” means the task was only seen and used by METR.
“public_problem” means the task is available somewhere online (so it could hav

...
Why I am not a successionist
by Nina Panickssery — 2025-05-04T19:08:52.645Z
Link: https://ninapanickssery.substack.com/p/aligned-ai-should-not-replace-humanity

Utilitarianism implies that if we build an AI that successfully maximizes utility/value, we should be ok with it replacing us. Sensible people add caveats related to how hard it’ll be to determine the correct definition of value or check whether the AI is truly optimizing it.

As someone who often passionately rants against the AI successionist line of thinking, the most common objection I hear is "why is your definition of value so arbitrary as to stipulate that biological meat-humans are necessary." This is missing the crux—I agree such a definition of moral value would be hard to justify.

Instead, my opposition to AI successionism comes from a preference toward my own kind. This is hardwired in me from biology. I prefer my family members to randomly-sampled people with similar traits. I would certainly not elect to sterilize or kill my family members so that they could be replaced with smarter, kinder, happier people. The problem with successionist philosophies is that they deny this preference altogether. It’s not as if they are saying "the end to humanity is completely inevitable, at least these other AI beings will continue existing," which I would understand. Instead, they are saying we should be happy with and choose the path of human extinction and replacement with "superior" beings.

That said, there’s an extremely gradual version of human improvement that I think is acceptable, if each generation endorses and comes of the next and is not being "replaced" at any particular instant. This is akin to our evolution from chimps and is a different kind of process from if the chimps were raising llamas for meat, the llamas eventually became really smart and morally good, peacefully sterilized the chimps, and took over the planet.

Luckily I think AI X-risk is low in absolute terms but if this were not the case I would be very concerned about how a large fraction of the AI safety and alignment community endorses humanity being replaced by a sufficiently aligned and advanced AI, and would prefer this to a future where our actual descendants spread over the planets, albeit at a slower pace and with fewer total objective "utils". I agree that if human extinction is near-inevitable it’s worth trying to build a worthy AI successor, but my impression is that many think the AI successor can be actually "better" such that we should choose it, which is what I’m disavowing...
Overview: AI Safety Outreach Grassroots Orgs
by Severin T. Seehrich — 2025-05-04T17:39:10.352Z

We’ve been looking for joinable endeavors in AI safety outreach over the past weeks and would like to share our findings with you. Let us know if we missed any and we’ll add them to the list.

For comprehensive directories of AI safety communities spanning general interest, technical focus, and local chapters, check out

https://www.aisafety.com/communities

and

https://www.aisafety.com/map

. If you're uncertain where to start,

https://aisafety.quest/

offers personalized guidance.

ControlAI

ControlAI started out as a think tank. Over the past months, they developed a theory of change for how to prevent ASI development (“

Direct Institutional Plan

”). As a pilot campaign they cold-mailed British MPs and Lords to talk to them about AI risk. So far, they talked to 70 representatives of which 31 agreed to publicly stand against ASI development.

Control AI is also supporting grassroots activism: On

https://controlai.com/take-action

, you can find templates to send to your representatives yourself, as well as guides for how to constructively inform people about AI risk. They are also reaching out to influencers and

supporting content creation

.

While they are the org on this list whose theory of change and actions we found most convincing, so far, they are still at the start of building infrastructure that would allow them to take in considerable numbers of volunteers. We expect them to react positively anyways if you reach out to them with requests for talks, training or similar. You can join the Control AI

Discord here

.

ControlAI is currently

hiring

!

EncodeAI

EncodeAI is an organization of high school and college students that addresses all kinds of AI risks. Their past endeavors and successes include a bipartisan event advocating for anti-deepfake laws, and co-sponsoring

SB 1047

, California’s landmark AI safety legislation that would, if passed, have been a tremendous contribution to AI existential safety.

You can find an overview of their past activities

here

and join their local chapters or start a new one

here

.

PauseAI

PauseAI is a community-focused organization dedicated to AI safety activism. Their primary aim is to normalize discussions about AI existential risk and advocate for a pause in advanced AI development. They contact policymakers, influencers and experts, organize protests, hand out leaflets, do

tabling

...
The Power Users We Forgot: Why AI Needs Them Now More Than Ever
by Anthony Fox — 2025-05-04T17:23:09.073Z

The People Who Were Right Too Early

There’s a quiet frustration shared by many in this community.

You raised concerns about AI before it was cool.
You explored edge cases before they were dangerous.
You built frameworks to understand systems that didn’t yet exist.

And for years, you were ignored.

Now the world is listening—but not to you.
The language of alignment, safety, and control is everywhere—
…but the incentives are still misaligned, and the people driving policy are often just chasing headlines.

Worse, the platforms and systems you once shaped—through code, critique, and deep analysis—have become closed-off, gamified, and optimized for scale over substance.

This isn’t new.

It happened to musicians.
It happened to developers.
And now, it’s happening in AI.

Power users—the ones who push systems to their limits by actually using them—are being sidelined again.

But if those users are ignored, we lose our most important feedback loop.
We don’t just lose control of the technology.
We lose contact with reality.

What Is an AI Power User?

The term “AI power user” isn’t widely defined yet—but it needs to be.

We can borrow the concept from the early days of computing. A computer power user wasn’t a programmer or hardware engineer. They were the people who pushed machines to the edge of their capabilities—automating tasks, scripting shortcuts, bending off-the-shelf tools to fit real workflows.

They didn’t build the system.
They made it do more than it was designed for.

An AI power user is the same kind of person, in a new context.

They’re not researchers. Not alignment theorists.
They’re people trying to get real work done with AI.

That includes:
- Writers structuring arguments or summarizing hours of transcripts
- Analysts exploring huge datasets with natural language queries
- Entrepreneurs gluing together GPT and Zapier to build microservices
- Designers prototyping content in minutes instead of hours

They don’t care about the model’s elegance. They care whether it does the job.

Power users aren’t hired to “test” systems—they stress them by using them.
And when something breaks, they find a workaround, file a complaint, or switch tools.

Their value isn’t academic.
It’s operational.

They reveal what no benchmark can:
Where the tool actually fails people.

The Power Users We Left Behind

This has happened before.

Musicians: From Creators to Content
Musicians once drove innovation in digital audio—DAWs, ...
Fake AI lawsuits to drive links
by Yair Halberstadt — 2025-05-04T16:53:52.872Z
Link: https://www.rationalistjudaism.com/p/the-other-sharks-out-there

Someone sent me this and I thought it fairly interesting:

TLDR: Fake AI generated law companies are searching for unattributed images on the internet, claiming they own them, and sending emails asking the user to add a link to their website. The aim is to increase their ranking in Google search to drive traffic to their (AI generated) website.

Yet another example of how the world is going to get steady wierder as signals we used to use for somebody putting in effort into something no longer apply.