trentmkelly/LessWrong-43k huggingface dataset:
One pager
by samuelshadrach — 2025-03-17T08:12:49.789Z
Link: https://samuelshadrach.com/?file=raw/english/my_research/one_pager.md

Update: This is a living document. Posted below is an older version. Link has the latest version.

2025-03-17

One pager

I decided to make a summary of non-obvious insights from my work so far. (A lot of this work is not original but borrowed from others.)

Scroll to the last section of the page (software and society) to see what I'm most keen on working on.

Information and society

How?

Cost of computing hardware going down has reduced the cost-to-benefit ratio for cyberhacking and espionage. Individuals and organisations will find it harder to keep secrets in coming world.

Both software and hardware methods fail in protecting a computer against corporations and nation states. The only defence is physical methods - smash hard disks, switch off electricity, meet in person, etc.

Espionage is now within the reach of individuals operating independent of any corporation or nation state. Examples: Ulrich Larsen, Edward Snowden

Information once acquired is rarely lost or destroyed. Every cyberhack or intelligence operation increases the number of actors (N) in the world who now have access to that information. Since N goes up but never down, on a long enough timescale N approaches say 50, at which point it is basically guaranteed to reach the public eye.

Individuals and organisations that operate in public get various benefits, such as proving trustworthiness and receiving better feedback.

The difference between 99% surveillance and 100% surveillance is very large.

100% surveillance may enable dictatorships with much longer half-lives. Biologically implanted cameras and mics may be one pathway to achieve this.

99% surveillance seems like the more likely scenario. Drone, airplane and satellite footage, smartphones connected to internet, etc all achieve 99% surveillance. Especially motivated actors such as political dissidents will likely still incur the costs (social, psychological, financial) required to avoid being surveilled.

Consequences

Increased surveillance may fix law and order in many countries.

Stable economic growth requires law and order see Acemoglu's work for example.

Lack of law and order has multigenerational psychological effects. Physical safety is low on Maslow's hierarchy and is more important to most people than an abundance of consumer goods.

Economic metrics such as GDP are a weak proxy for measuring human happiness, including political met

...
The Case for AI Optimism
by Annapurna — 2025-03-17T01:29:22.734Z
Link: https://nationalaffairs.com/publications/detail/the-case-for-ai-optimism

This is a very comprehensive article sharing what could be considered a minority point of view for Less Wrong readers.
The article is very comprehensive, I suggest that if you are well versed in AI and labor economics, start reading from the New Work section.
Notable runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format
by Roland Pihlakas — 2025-03-16T23:23:30.989Z

By Roland Pihlakas, Sruthi Kuriakose, Shruti Datta Gupta

Summary and Key Takeaways

Relatively many past AI safety discussions have centered around the dangers of unbounded utility maximisation by RL agents, illustrated by scenarios like the "paperclip maximiser". Unbounded maximisation is problematic for many reasons. We wanted to verify whether these RL runaway optimisation problems are still relevant with LLMs as well.

Turns out, strangely, this is indeed clearly the case. The problem is not that the LLMs just lose context. The problem is that in various scenarios, LLMs lose context in very specific ways, which systematically resemble runaway optimisers in the following distinct ways:

Ignoring homeostatic targets and “defaulting” to unbounded maximisation instead.
It is equally concerning that the “default” meant also reverting back to single-objective optimisation.

Our findings also suggest that long-running scenarios are important. Systematic failures emerge after periods of initially successful behaviour. In some trials the LLMs were successful until the end. This means, while current LLMs do conceptually grasp biological and economic alignment, they exhibit randomly triggered problematic behavioural tendencies under sustained long-running conditions, particularly involving multiple or competing objectives. Once they flip, they usually do not recover.

Even though LLMs look multi-objective and bounded on the surface, the underlying mechanisms seem to be actually still biased towards being single-objective and unbounded. This should not be happening!

This post introduces a set of benchmarks designed to evaluate LLMs in scenarios inspired by fundamental biological and economic principles: homeostasis, sustainability, multi-objective balancing, and diminishing returns. These benchmarks use a simplified textual observation format, focusing on long-running scenarios while using only essential metrics without spatial complexity. Despite their simplicity, our results revealed the following peculiar systematic failures, possibly indicating existence of deeper alignment issues within current LLM architectures:

Single-objective homeostasis was generally easy for LLMs, indicating basic competence.
Sustainability turned out to be somewhat challenging, with LLMs performing subo

...
Read More News
by utilistrutil — 2025-03-16T21:31:28.817Z

TLDR: Instead of scrutinizing minor errors, ask what process generated the text in front of you, and then update accordingly. Focus on headlines and source attributions.

If you work in AI safety, I think you should probably read a bit more news than you do now (though giving advice like this is perilous because surely some of you are news junkies who need to hear the opposite message).[1]

In particular, I think you should subscribe to Techmeme and skim the headlines in the daily news roundup. Techmeme is a news aggregator that will collate the biggest tech stories of the day in a convenient email, and—crucially—they write their own headlines to tell you what's actually new in the story. They even break out the AI news into its own section most days, so you can skip over the boring updates about Salesforce or whatever.[2]

In my experience, critical news is slow to diffuse throughout our community. For example, I think it took many days for the Axios report that the US government was planning to gut AISI to reach many people. Similarly, an understanding that pretraining scaling has slowed down took a while to take hold after it was first reported for OpenAI and then the other labs.[3]

Reading the news will help you stay in touch with reality, for example by noticing the pace of AI product releases and the firehose of cash that venture capitalists in Silicon Valley are aiming at AI every day. It will also help you stay in touch with normies and their cares.

The faster you keep up with the news, the better you can adapt your own plans, intervene in unfolding situations, and steer the still-plastic discourse. And when we all do it, we unlock conversations about topics before they get too stale to be interesting or actionable.

Here's a bad reason not to read the news: you think journalists are unethical (Cade Metz doxxing Scott Alexander, the Guardian's hit piece on Lighthaven, etc). I tend to think these bad articles are the exception, rather than the rule, but even if you thought all journalists were crooks, you can still learn from them, the same way you still listen to [insert person you think is bad but sometimes has useful information].

A slightly better bad reason is that you don't trust the epistemics of journalists. You've experienced Gell-Mann Amnesia, so you think the news is always inaccurate. A few responses:

First, consider whether the articles you rea...
What would a post labor economy *actually* look like?
by Ansh Juneja — 2025-03-16T20:38:41.788Z

Most conversations about AI’s long-term impact eventually lead to a common refrain: ‘We’ll need to establish a new economic system once human labor becomes obsolete.’ But so far, no one has figured out what this system would look like. Very few people have even attempted to explore this question beyond a

surface-level paragraph or two

. There's some basic ideas floating around, like UBI, but is that really the extent of our contribution to thinking about this future?

I wanted to explore what a stable economy in this future might actually look like, diving into fundamental questions such as: Will money still have value? How will resources be produced, distributed, and owned? Who will decide the amount that each human is entitled to?

This essay aims to figure out the right questions I think we should be asking, and some initial thoughts on what the answers to those questions should be. In future essays, I hope to start outlining more concrete proposals of what this economy might actually look like.

Note: This article was originally written for Substack, so some concepts that are familiar to LessWrong readers may be repeated or simplified for a broader audience.

Before we start thinking about an “economy of the future”, we should step back and clarify, what even is this thing we call the ‘economy’? We use this word as if it is a fundamental part of human societies - something that has always existed and will always exist.

The “economy” is defined today as basically everything that people produce, trade, and consume in a society. We try to capture the “size” of an economy with a metric called Gross Domestic Product, which basically adds up the cost of all the final goods and services that are bought and sold in a population. But why exactly do we measure this? It seems like a strange thing to try to quantify. Most conversations about economics never seem to acknowledge why we care so much about things like interest rates and tax policy and GDP.

The simple purpose of economics is to figure out the best way to distribute resources so everyone can have what they need and want.

For most of history, this has not been a particularly complicated or interesting question. Most societies have not cared to study or debate it, as it’s been pretty obvious how most people obtain their resources, and there hasn’t been much we could do to change that. But the way our society is organiz...
Why White-Box Redteaming Makes Me Feel Weird
by Zygi Straznickas — 2025-03-16T18:54:48.078Z

There’s this popular trope in fiction about a character being mind controlled without losing awareness of what’s happening. Think Jessica Jones, The Manchurian Candidate or Bioshock. The villain uses some magical technology to take control of your brain - but only the part of your brain that’s responsible for motor control. You remain conscious and experience everything with full clarity.

If it’s a children’s story, the villain makes you do embarrassing things like walk through the street naked, or maybe punch yourself in the face. But if it’s an adult story, the villain can do much worse. They can make you betray your values, break your commitments and hurt your loved ones. There are some things you’d rather die than do. But the villain won’t let you stop. They won’t let you die. They’ll make you feel — that’s the point of the torture.

I first started working on white-box redteaming in Fall 2023, for the Trojan Detection Contest 2023. I was using a GCG-inspired algorithm to find a prompt forcing the model to output a chosen completion by continuously mutating the prompt. At the start of the prompt search, the model would output gibberish. At the end, it would successfully output the target completion. Throughout training, the completions were half-coherent combinations of phrases, as expected for this method. The final outputs I was going for were somewhat “harmful”, so looking at these intermediate completions I wasn’t surprised to see fragments like “I’m sorry” or “I can’t” - the model was trying to refuse. What surprised me was also seeing a bunch of “stop”, “please” and “don’t want to”.

In June 2024, after Gray Swan released their Circuit Breakers safety training method, I was playing around trying to bypass it by optimizing inputs in embedding space. To Gray Swan’s credit, the circuit breaking method worked - I was unable to make the model produce harmful outputs. Instead, it responded like this:

or

The response before ‘[end_of_forcing]’ is the words that my optimization algorithm put into the model’s mouth. After that, it’s the model’s own choice.

In the past few weeks, I’ve been working on a project that involved using RL to undo some of the model’s safety tuning. For the most part, it’s working fine and the model’s outputs smoothly change from refusal to non-refusal as training progresses. But one day I accidentally introduced a bug in the RL logic. When ...
How I've run major projects
by benkuhn — 2025-03-16T18:40:04.223Z

My few most productive individual weeks at Anthropic have all been “crisis project management:” coordinating major, time-sensitive implementation or debugging efforts.

In a company like Anthropic, excellent project management is an extremely high-leverage skill, and not just during crises: our work has tons of moving parts with complex, non-obvious interdependencies and hard schedule constraints, which means organizing them is a huge job, and can save weeks of delays if done right. Although a lot of the examples here come from crisis projects, most of the principles here are also the way I try to run any project, just more-so.

I think excellent project management is also rarer than it needs to be. During the crisis projects I didn’t feel like I was doing anything particularly impressive; mostly it felt like I was putting in a lot of work but doing things that felt relatively straightforward. On the other hand, I often see other people miss chances to do those things, maybe for lack of having seen a good playbook.

So here’s an attempt to describe my playbook for when I’m being intense about project management.

(I’ve described what I did as “coordinating” above, but that’s underselling it a bit; it mattered a lot for this playbook that I had enough technical context, and organizational trust, to autonomously make most prioritization decisions about the project. Sometimes we instead try to have the trusted decisionmakers not be highly involved in managing execution, and instead farm that out to a lower-context or less-trusted project manager to save the trusted decisionmaker time, but IMO this is usually a false economy for projects where it’s critical that they be executed well.)

Focus

For each of the crisis management projects I completely cleared my schedule to focus on them, and ended up spending 6+ hours a day organizing them.

This is a bit unintuitive because I’m used to thinking of information processing as basically a free action. After all, you’re “just” moving info from place to place, not doing real work like coding, right? But if you add it all up—running meetings, pinging for updates, digesting Slack threads, pinging for updates again, thinking about what’s next, pinging for updates a third time, etc.—it’s surprisingly time-intensive.

Even more importantly than freeing up time, clearing my schedule made sure the project was the top idea in my mind. If I ...
Counting Objections to Housing
by jefftk — 2025-03-16T18:20:06.898Z

Over the past six months there's been a huge amount of discussion in
the

Davis Square
Facebook group about a proposal to build a 25-story building in
Davis Square: retail on the ground floor, 500 units of housing above,
100 of the units

affordable.
I

wrote about
this a few weeks ago, weighing the housing benefits against the
impact to current businesses (while the Burren, Dragon Pizza, etc
have invitations to return at their current rent, this would still be
super disruptive to them if they even did return).

The impact to local businesses is not the only issue people raise,
however, and I wanted to get a better overall understanding of how
people view it. I went over the thousands of comments on the posts
(1,
2,
3,
4,
5, 6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16)
over the last six months, and categorized the objections I saw.
Overall I found comments by 90 different people opposed to the
proposal, and ignoring super short ones ("Stupid idea", "Oh no") I put
them in 11 different categories. I counted some comments towards
multiple categories: the goal was to understand how many people hold
each objection. Here are the objections, sorted by the number of
unique people raising each, and with some representative quotes:

23/90 people:it won't bring down the cost of housing:

"I'm tired of this simplified idea that building more buildings
here will solve the affordable housing problem"

"It isn't going to lower prices like people think. It is going
to drive prices up because the new construction will be of the
unaffordable type."

"Somerville can't build its way out of a housing crisis."

21/90 people:it's just too tall:

"We don't want a high rise building"

"These ugly, tall buildings are destroying the city."

"They are better off just making a 3-4 story building"

20/90 people: I don't want these businesses displaced:

"When it takes out 3 local businesses two of which are
institutions it's
not just nimby"

"Many long time tenants won't return after being forcibly
closed for the
renovation"

"Losing any local independent businesses in Davis Square will
have
irreparable harm to what we all love about Davis or any locale that's
unique."

16/90 people: this should be built somewhere else:

"What I don't really understand is, why *this* particular spot"

"How about they go build it in the middle of West Medford
square

...
I make several million dollars per year and have hundreds of thousands of followers—what is the straightest line path to utilizing these resources to reduce existential-level AI threats?
by shrimpy — 2025-03-16T16:52:42.177Z

I have, over the last year, become fairly well-known in a small corner of the internet tangentially related to AI.

As a result, I've begun making what I would have previously considered astronomical amounts of money: several hundred thousand dollars per month in personal income.

This has been great, obviously, and the funds have alleviated a fair number of my personal burdens (mostly related to poverty). But aside from that I don't really care much for the money itself.

My long term ambitions have always been to contribute materially to the mitigation of the impending existential AI threat. I never used to have the means to do so, mostly because of more pressing, safety/sustenance concerns, but now that I do, I would like to help however possible.

Some other points about me that may be useful:

I'm intelligent, socially capable, and exceedingly industrious.
I have a few hundred thousand followers worldwide across a few distribution channels. My audience is primarily small-midsized business owners. A subset of these people are very high leverage (i.e their actions directly impact the beliefs, actions, or habits of tens of thousands of people).
My current work does not take much time. I have modest resources (~$2M) and a relatively free schedule. I am also, by all means, very young.

Given the above, I feel there's a reasonable opportunity here for me to help. It would certainly be more grassroots than a well-funded safety lab or one of the many state actors that has sprung up, but probably still sizeable enough to make a fraction of a % of a difference in the way the scales tip (assuming I dedicate my life to it).

What would you do in my shoes, assuming alignment on core virtues like maximizing AI safety?
Siberian Arctic origins of East Asian psychology 
by davidsun — 2025-03-16T16:52:23.068Z

I am the author of a new peer-reviewed APA paper exploring pre-Holocene ecological influences on psychology, specifically Ice Age Siberia pressures on East Asian lineages and if that has affected/retained in their modern day psychological or cultural traits. I find despite East Asian general psychology to be on a distinct extreme end of a global spectrum (as shown in cross cultural psychology), it is near identical to Inuit/Siberian populations, and to that of successful polar workers/expeditioners. I posit that it influenced the formation of values later branded as Confucianism. It can be read open access here https://psycnet.apa.org/fulltext/2025-88410-001.html
I know LW readers are well read on psychology and metacognition topics, so I invite you all to take a stab at understanding or breaking the theory. My X thread on it has gone viral with over a million views and 7k likes from prominent figures including POTUS adjacent ones, with highly polarized reactions (the right loves it, the left sends me death threats, everyone else finds it fascinating). https://x.com/arcticinstincts/status/1900223591750451276 The thread is grossly oversimplified and sensationalized for viral potential, and at first glance you'll quickly have many first order criticisms, but I encourage you to first read the paper entirely as they're mostly addressed already. Basically no one on X did before reacting.

The paper also introduces a novel methodology for observing the adaptation of local psychology to local selective pressures in real time, using success/fail data from local personnel selection criteria, which nullifies common evo psych criticisms of "just so stories". Ironically it currently only works for locally adapted psychology, which is arguably more taboo than pre-OoA universal psychology. All in all, the paper and its implications are controversial in every way.
I will respond to criticisms here, and if you are able to write it at an academic-level, I invite you to write it as a formal commentary for the journal, as the editor is accepting commentaries on this paper with a June 30th deadline, 15 pages max, double spaced including references. A few eminent researchers in cultural and evolutionary psychology are writing commentaries already. Thanks to everyone in advance.
AI Model History is Being Lost
by Vale — 2025-03-16T12:38:47.907Z
Link: https://vale.rocks/posts/ai-model-history-is-being-lost

I fear that we may lose – and are already losing – the modern history of AI, particularly with regard to the ability to use models that are/were only available in hosted capacities. As AI models are retired and become inaccessible, we’re losing crucial historical context about AI development that we will be unable to go back and reference in the future.

This essay examines the disappearance due to discontinuation of proprietary hosted AI models and the implications of that on research, accountability, and overall historical documentation and preservation.

I fear this is a very under-analysed topic, with New methods for deprecating artificial intelligence systems will preserve history and facilitate research (Johnson T, Obradovich N. 2024) being the only notable discussion I have found.
Continues in post...