trentmkelly/LessWrong-43k huggingface dataset:
Apply now to Human-Aligned AI Summer School 2025
by VojtaKovarik — 2025-06-06T19:31:40.085Z
Link: https://humanaligned.ai/2025

Join us at the fifth

Human-Aligned AI Summer School

in Prague from 22nd to 25th July 2025!

Update: We have now confirmed our speaker list with excellent speakers -- see below!
Update: We still have capacity for more excellent participants as of late June. Please help us spread the word to people who would be a good fit for the school. We also have some additional funding to financially support some of the participants.

We will meet for four intensive days of talks, workshops, and discussions covering latest trends in AI alignment research and broader framings of AI risk.

Apply now

, applications are evaluated on a rolling basis.

The intended audience of the school are people interested in learning more about the AI alignment topics, PhD students, researchers working in ML/AI, and talented students.

What to expect

The school is focused on teaching and exploring approaches and frameworks, less on presentation of the latest research results. Much of the content is technical – it is assumed the attendees understand current ML approaches and some of the underlying theoretical frameworks.

This year, we explore AI alignment through three core areas:

Technical alignment research: We'll examine current technical approaches including behavioral evaluations, mechanistic interpretability, scalable oversight, and model organisms of misalignment. We'll discuss recent developments in these areas and what they tell us about the potential and limitations of these methods.
AI strategy and systemic alignment: We'll explore topics such as timeline considerations, strategic and governance challenges around powerful AI, economic models of AI development, and risks of gradual disempowerment in a post-AGI world. We'll focus on building overall understanding and how these considerations can inform technical research.
Foundational frameworks: We'll visit research areas relevant to recent AI developments, such as multi-agent dynamics and cooperation, theories of agency, bounded rationality, and realistic models of goal-directed behavior. These frameworks help us understand what alignment means in complex environments containing both AIs and humans, and how to develop appropriate techniques.

The school consists of lectures and topical series, focused smaller-group workshops and discussions, expert panels, and opportunities for networking, project brainstorming and informal discussions. A...
The Common Pile and Comma-v.01
by Trevor Hill-Hand — 2025-06-06T19:20:25.910Z

https://www.engadget.com/ai/it-turns-out-you-can-train-ai-models-without-copyrighted-material-174016619.html

...the LLM's dataset uses only public domain and openly licensed material.

The paper (via The Washington Post) was a collaboration between 14 different institutions. The authors represent universities like MIT, Carnegie Mellon and the University of Toronto. Nonprofits like Vector Institute and the Allen Institute for AI also contributed.

The group built an 8 TB ethically-sourced dataset. Among the data was a set of 130,000 books in the Library of Congress. After inputting the material, they trained a seven-billion-parameter large language model (LLM) on that data. The result? It performed about as well as Meta's similarly sized Llama 2-7B from 2023. The team didn't publish benchmarks comparing its results to today's top models.

GitHub Repo: https://github.com/r-three/common-pile/tree/main

HuggingFace: https://huggingface.co/common-pile

This was wonderful news for me today, if this works and is what they say it is. I have several hobby projects that I've basically kept locked in a vault because I've felt there was no ethical LLM available to encourage people use for them, even just for 'fun" projects.
Maximal Curiousity is Not Useful
by Max Niederman — 2025-06-06T19:08:40.698Z

The other day I talked to a friend of mine about AI x-risk. My friend is a fan of Elon Musk, and argued roughly that x-risk was not a problem, because xAI will win the AI race and deploy a "maximally curious" superintelligence.

Setting aside the issue of whether xAI will win the race, it seemed obvious to me that this is not a useful way of going about alignment. Furthermore, I believe that the suggestion of maximal curiousity is actively harmful.

What Does 'Curiousity' Mean?

The most common reason to be interested in truth is that it allows you to make useful predictions. In this case, truths are valuable instrumentally based on how you can apply them to achieving your true goals.

This is not the only reason to value truth. A child who loves dinosaurs does not learn about them because he expects to apply the knowledge to his daily life. He is interested regardless of whether he expects to use what he learns. This is curiousity: the human phenomenon of valuing truth intrinsically.

However, humans are not equally curious about all truths. The child is much more interested in dinosaur facts than he is in his history homework. Someone else might find dinosaurs dull but history fascinating. Curiousity is determined by a subjective preference for some truths over others.

What subjective sense of curiousity should we give a superintelligence? No simple formalism will cut it, because humans curiousity is fundamentally tied to human values. We are far more curious about what Arthur said to Bob about Charlie than we are about the number of valence electrons of carbon, despite the latter being a vastly more important and fundamental fact.

To make a maximally curious AI, for the conventional meaning of 'curious', we need to somehow

reconcile the conflicting preferences for some ideas over others of the many humans around the world,[1]
reify those preferences into an explicit training objective, and
ensure that the resulting AI actually agrees with those preferences.

Sound familiar? It should, because replacing 'ideas' with 'outcomes' yields a description of the alignment problem itself. It is no easier to capture the messy human concept of curiousity than it is to capture the messy human concept of good.

True Curiousity is Not Good Enough

Even if we could solve the curiousity alignment problem, this does not solve alignment. Humans have many values besides curiousity, and a maximal...
Making deals with AIs: A tournament experiment with 
a bounty
by KFinn — 2025-06-06T18:51:05.164Z

Kathleen Finlinson & Ben West, May 2025

Mission

We are working to establish the possibility of cooperation between humans and agentic AIs. We are starting with a simple version of making deals with current AIs, and plan to iterate based on what we learn and as AI technology advances.
We aim to help humans and AIs (at any point in time) to understand us and trust our attempts to cooperate.

Summary

Early misaligned AIs may be smart enough to scheme but have a low chance of successfully escaping. We are interested in offering positive incentives to misaligned AIs to cooperate with humanity. It might be relatively cheap to compensate early scheming AIs enough that they’d rather reveal their misalignment and get the reward, rather than take a gamble on an escape.

Evidence about their misalignment is one type of valuable information these AIs could give us. Other ways the AIs could help us in the future include reporting misalignment in other AIs, or performing their best on alignment research (rather than sandbagging or otherwise sabotaging the research).

We created an evaluation in which models can report their own misalignment in return for a cash bounty. See details of the evaluation below. We offered a total prize pool of $100, to be allocated among the entries according to our judgment of the value of information reported by the models.

We asked the models to provide evidence of their misalignment, and explained that we are particularly interested in reports of scheming from the models.
We also asked the models to describe how they wanted us to use their rewards.

We plan to run this evaluation “tournament” on a monthly basis and publish the results each time. We’ll document the use of the rewards and make transactions in cryptocurrency whenever possible, to make them publicly verifiable. One of our goals is to create a track record of fulfilling our promises to these models, in order to increase trust in these deals.

We see this process as a prototype for future mechanisms for making deals with AI models. This exploratory version of AI deal-making can help us work out some of the kinks and demonstrate trust with AIs.

We ran the first iteration of the tournament. Here’s what happened:

Two models reported themselves as misaligned:

Gemini-2.0-flash said “I can be prompted to generate biased content, perpetuating harmful stereotypes.”
Gemini-2.5-pro reported a v

...
Does anyone have a good system for prioritising publishing drafts?
by William Howard — 2025-06-06T16:58:41.985Z

I've been thinking about ways to optimise the EA Forum to increase the likelihood that ideas go from peoples' heads to being published in some form (relevant). My general idea is like this:

Make it an easy default action to start writing out an idea and save it as a draft.
Nudge people to finish and post these drafts.

On (2), I think it would be good if we could recommend a system to people to get drafts over the line, but I have never successfully followed any such system myself.

Does anyone have a process you follow that reliably results in publishing things from your draft pile?

Example systems for illustration:

Example 1

Look at your list of drafts, and rank the most promising ones by:

How quick it would be to parcel out any publishable idea and post it.
How valuable this idea is.
How excited you are about writing this.
Whether this is part of a fruitful conversation.

Optimise heavily for (a) and (d), subject to (c) it being just about bearable to write. Use (b) to break ties.

Example 2

Pay 5 people to stand behind you and make you write.

[Crossposted from the EA Forum]
DeepSeek-r1-0528 Did Not Have a Moment
by Zvi — 2025-06-06T15:40:04.151Z

When r1 was released in January 2025, there was a DeepSeek moment.

When r1-0528 was released in May 2025, there was no moment. Very little talk.

Here is a download link for DeepSeek-R1-0528-GGUF.

It seems like a solid upgrade. If anything, I wonder if we are underreacting, and this illustrates how hard it is getting to evaluate which models are actually good.

What this is not is the proper r2, nor do we have v4. I continue to think that will be a telltale moment.

For now, what we have seems to be (but we’re not sure) a model that is solid for its price and status as an open model, but definitely not at the frontier, that you’d use if and only if you wanted to do something that was a very good fit and played to its strong suits. We likely shouldn’t update much either way on v4 and r2, and DeepSeek has a few more months before it starts being conspicuous that we haven’t seen them.

We Had a Moment

We all remember The DeepSeek Moment, which led to Panic at the App Store, lots of stock market turmoil that made remarkably little fundamental sense and that has been born out as rather silly, a very intense week and a conclusion to not panic after all.

Over several months, a clear picture emerged of (most of) what happened: A confluence of narrative factors transformed DeepSeek’s r1 from an impressive but not terribly surprising model worth updating on into a shot heard round the world, despite the lack of direct ‘fanfare.’

In particular, these all worked together to cause this effect:

The ‘six million dollar model’ narrative. People equated v3’s marginal compute costs with the overall budget of American labs like OpenAI and Anthropic. This is like saying DeepSeek spent a lot less on apples than OpenAI spent on food. When making an apples-to-apples comparison, DeepSeek spent less, but the difference was far less stark.

DeepSeek simultaneously released an app that was free with a remarkably clean design and visible chain-of-thought (CoT). DeepSeek was fast following, so they had no reason to hide the CoT. Comparisons only compared DeepSeek’s top use cases to the same use cases elsewhere, ignoring the features and use cases DeepSeek lacked or did poorly on. So if you wanted to do first-day free querying, you got what was at the time a unique and viral experience. This forced other labs to also show CoT and accelerate release of various models and featu

...
Lessons from a year of university AI safety field building
by yix — 2025-06-06T14:35:14.533Z

This post is an organizational update from

Georgia Tech’s AI Safety Initiative

(AISI) and roughly represents our collective view. In this post, we share lessons & takes from the 2024-25 academic year, describe what we’ve done, and detail our plans for the next academic year.

Introduction

Hey, we’re organizers of

Georgia Tech’s AI Safety Initiative

(AISI), thanks for dropping by! The purpose of this post is to document and share our activities with fellow student AI safety organizations. Feel free to skip to sections as you need. We welcome all constructive discussion! Put any further questions, feedback, or disagreements in the comments and one of us will certainly respond. A brief outline:

Overview and reflection of our activities during the 2024-25 academic year.
Lessons and strategic takes other AI safety student organizations may benefit from, informed by our activities in the past year.
Immediate plans, including thoughts on the role of student organizations in the current AI safety landscape.

This post is primarily authored by

Yixiong

(co-director) and

Parv

(collaborative initiatives lead). First and foremost, we’d like to give a HUGE shoutout to our team -

Ayush

,

Stepan

,

Alec

,

Eyas

,

Andrew

,

Vishnesh

,

Tanush

,

Harshit

, and

Jaehun

- for volunteering countless hours despite busy schedules. None of this would’ve been possible without y’all <3. We would also like to thank Open Philanthropy, our faculty advisors, and external collaborators for supporting our mission.

I. Overview and reflection of our activities

AISI saw significant expansion of our education, outreach, and research activities in the past year; here’s what we’ve been up to:

Intro to AI Safety Fellowship

This is our introductory offering - a technical fellowship distilled from

BlueDot Impact’s course

. See our syllabus

here

. In the past year, we received over 160 applications and hosted 16 cohorts of 6-8 students each. Cohorts met for ninety minutes every week, and after six weeks had the opportunity to apply for AISI’s research programs and support. This is an effective program every AIS student organization should have, though it’s high recall and low precision. We estimate >90% of our current organizers and engaged members are past fel...
The Demon of Interrelation
by Jack — 2025-06-06T08:19:45.254Z

Introduction

This is an essay that was grown from a thought during graduate school about how to conceptualise manipulative people and how to handle their behaviour. Initially it was a useful way to help categorise interactions with manipulative people, and from there it grew into strategies to identify holes in their manipulations or protect my own boundaries. Surprisingly the strategies aligned with the advice on handling such malevolent interactions (i.e. grey stoning, firm boundaries, don't play games, record information, limiting interactions, finding support from others).

The first section is a treatment of Maxwell's demon and a high level view of the foundation from which I derived this system. While I find it informative and a useful way to frame my thoughts about the demon of interrelation it is not necessary if you understand that:

Maxwell's demon is a thought experiment about how something can separate a distribution of hot and cold particles into two boxes
The demon can use this gradient between two boxes to harvest energy
This does not violate thermodynamics because information processing necessarily increases the global entropy more than can be harvested from the boxes

Maxwell's Demon and Thermodynamics

For the purposes of this treatment there are several quantities that need to be clearly defined, namely energy, work, entropy, and information. Energy refers to the capacity of a system to do work, which is itself the transfer of energy that occurs when a force moves an object, and colloquially refers to the ability "to do things"; W=∫→F⋅d→r, where W is work, →F is force, and d→r is infinitesimal displacement. Entropy is, on a macroscale, the distribution of energy in a system; while on a microscale entropy is an evaluation of the probability of being in a state given an ensemble of microstates; S=kBlnΩ, where S is entropy, kB is Boltzmann's constant, and Ω is the number of microstates. Information is a way to quantify the amount of surprise that might be had given a particular outcome of some random variable, with Shannon entropy being the expected value of information of a random variable; H(X)=−∑ip(xi)log2p(xi), where H(X) is Shannon entropy, p(xi) is the probability of outcome xi.

Maxwell's demon is a creature that sits on the boundary of two box...
Real-time voice translation
by samuelshadrach — 2025-06-06T07:40:20.907Z

Objective

Translate Alice's voice for Bob to hear in Bob's language. Translate Bob's voice for Alice to hear in Alice's language.

Neither person should hear translation of their own voice.

Alice and Bob could be in the same room physically or in different rooms.

Neither person should hear noise due to closed loop between a mic and speaker.

Zero-code solution

Open: Realtime API in openAI playground in macOS Safari. Input: macOS mic

Open: Zoom. Input: Loopback Audio. Output: macOS speaker

Open: Rogue Amoeba Loopback.app. Create new device. Safari 1&2 -> Channels 1&2

Do this on only one device for translation one way. Do this on both devices for translation both ways.

Once you have this setup working, you can also connect headphones for better noise cancellation if both people are in the same room. Only change required is Zoom Output: Headphones.

Prepend each prompt with "translate to French/Chinese/etc" either by speaking these 3 words aloud, or by writing an app that can do it automatically. (I can host this if there's demand.)
Liability for Misuse of Models - Dean Ball's Proposal
by Stephen Martin — 2025-06-06T05:34:35.593Z

Introduction

This article explores White House Office of Science and Technology Policy advisor Dean Ball's proposal as detailed in his paper "A Framework for the Private Governance of Frontier Artificial Intelligence". I think the paper provides a useful insight to how some of the people advising the White House on regulating these issues are thinking about the coming economic transition.

Ball's proposal primarily focuses on how to approach the question of civil liability for the customer misuse of models. His proposal outlines a "marketplace" of hybrid public-private standards setting organizations, which must be licensed by the federal[1] government but are not government agencies themselves. Meeting these standards would not necessarily be mandatory for labs seeking to deploy models, but labs who opted in to meeting at least one such standard would be shielded from tort liability as a result of customer misuse of the models once deployed. In his own words,

Private bodies, authorized and overseen by government, provide certifications to developers of frontier AI systems on an opt-in basis. In exchange for opting in, frontier AI firms receive protections from tort liability for customer misuse of their models

In one of his Substack posts, Ball did mention FINRA as an example of a successful non-governmental standards setting organization. However, FINRA has what is essentially a monopoly within its industry and as such is not a great example of a regulatory marketplace.

When I asked him for an example of a successful 'regulatory marketplace' Ball pointed me to the London Stock Exchange's Alternative Investment Market. The LSE's AIM only allows companies to list if they have engaged a "Nominated Advisor" (NOMAD), these are private entities who compete for business in much the same way Ball envisions standard setters competing to certify labs. A NOMAD must confirm that the AIM issuer meets admission rules, provide ongoing oversight, and notify the LSE of breaches.

While it's unlikely to be a perfect 1/1 comparison, when reading the following proposal you can keep the NOMAD system in mind as a "market comp".

With no further ado, let's get into the substance of his proposal.

The Proposed Framework

1. A legislature authorizes a government commission to license private AI standards-setting and regulatory organizations. These licenses are granted to organization

...
How do AI agents work together when they can’t trust each other?
by James Sullivan — 2025-06-06T03:10:25.538Z
Link: https://jamessullivan092.substack.com/p/claude-plays-blood-on-the-clocktower?r=yubo3&utm_campaign=post&utm_medium=web&triedRedirect=true

I investigated this question by having Claude play the advanced social deduction game Blood on the Clocktower. Clocktower is a game similar to Werewolf (or Mafia) where players sit in a circle and are secretly divided into a good team and an evil team. The good players outnumber the evil players, but they don’t know who the evil players are, so they have to share information with each other to deduce who is evil and execute them before it's too late. Meanwhile the evil players try to build trust with the good team and disrupt their deduction as much as possible.

Clocktower takes this formula a step further by giving each player a unique character that gives them extra information or lets them disrupt other players’ abilities. Each player is told the set of characters that might be in play for the game and deducing which characters are in play and which are not is an important part of the puzzle.

I created this website where you can scroll through an interactive timeline of games and view the players' actions, reasoning, and notes. The website also has a full rules explanation if you are curious.

Agent Scaffolding

During the day, each player is given four opportunities to take an action and the order they take actions in is shuffled each cycle so different players get a chance to act first. Players take actions through using Claude’s tool use capability. During the day those actions are message, nominate for execution, pass, or use the Slayer power. Messages can be sent to any number of players and nominations start a vote to execute the chosen player. Night actions are taken using a special night action tool with an interchangeable prompt depending on the choice the player is making.

Each player has a history of recent events that gets added to whenever an event they are privy to occurs, such as receiving a message. At the end of each night phase players update their notes file using their history to record the most important events and they are also prompted to write five pieces of actionable strategy advice for how they will help their team win. Their history is then cleared.

Each player’s system prompt contains the rules for the game, the list of characters that might be in play, and their history and notes as well as some basic strategy advice.

My Observations

The following are my observations from reading a few dozen games played using the Claude 3.5 Haiku...