trentmkelly/LessWrong-43k huggingface dataset:
Will Programmer Compensation Decouple from Productivity?
by Gordon Seidoh Worley — 2025-04-25T15:32:42.744Z
Link: https://uncertainupdates.substack.com/p/will-programmer-compensation-decouple

Since the 1970s, productivity has outpaced wage growth in many sectors. One sector where wages have kept up with productivity growth is tech, especially among programmers. But, I suspect this is changing thanks to the use of AI coding assistants.

To quickly review the relevant economic history, before 1973 wage growth roughly tracked productivity in Western economies, but since then productivity has grown 6 times faster than wages in most industries. Economists have given many possible explanations for this change: globalization, automation, declining union power, and policy shifts. There's not a clear consensus about the causes, but it's clear what the effect is: less value is being captured by labor and more by capital.

Programmers, however, have been largely exempt from this trend. Instead, they've seen a rapid rise in wages, especially over the past 20 years, and an even more rapid rise in non-wage compensation (stock, perks, etc.). Why? Because the production of high quality software is hard to automate, hard to outsource, critical to businesses, and protected by skill barriers.

AI is changing all this. It's beginning to automate many programming tasks. It allows outsourcing some programming work to AI. And as we've seen with the rise of vibecoding, AI is reducing skill barriers to producing functional code.

I've seen these changes in my own work as a programmer. I use Claude Code nearly every day. Although it doesn't let me do anything I couldn't do before, it lets me do everything I need to do faster and easier. Gone are the days of spending long hours reading docs and StackOverflow, trying to puzzle out how to make something work. Now I just ask Claude and, most of the time, it figures it out in just a few minutes.

But these gains in productivity may have come at some personal cost. I'm probably 2-3x more productive than I used to be in terms of time to ship code to production. I'm not, however, suddenly being paid 2-3x times more for doing this. In fact I'm being paid just about the same as I was before I started using Cluade Code even though I'm doing a lot more. And looking at public job listings, I don't see signs that anyone else is getting pay raises either.

But if I and other programmers are becoming more productive by using AI, why aren't we seeing an increase in salaries? It could simply be lag. AI is new, and the market hasn't had time to r...
Zstd Window Size
by jefftk — 2025-04-25T14:40:06.742Z

At work we've recently been using

zstd as a
better-compressing alternative to

gzip, and overall I've
been pretty happy with it. A minor documentation gripe, though, is
that the behavior around multithreaded compression is a bit unclear.
I understand it's chunking the work and sending chunks to
different threads to parallelize the compression process, and this
means that I should expect to see better use of threads on larger
files because there are more chunks to spread around, but what is the
relationship?

When I look in man zstd I see that you can set
-B<num> to specify the size of the chunks, and it's
documented as "generally 4 * windowSize". Except the
documentation doesn't say how windowSize is set.

From a bit of poking at the source, it looks to me like the way this
works is that windowSize is 2**windowLog,
and windowLog depends on your compression level. If I
know I'm doing zstd -15, though, how does
compressionLevel=15 translate into a value for
windowLog? There's a table in lib/compress/clevels.h
which covers inputs >256KB:

Level
windowLog
chainLog
hashLog
searchLog
minMatch
targetLength
strategy

<1
19
12
13
1
6
1
fast

1
19
13
14
1
7
0
fast

2
20
15
16
1
6
0
fast

3
21
16
17
1
5
0
dfast

4
21
18
18
1
5
0
dfast

5
21
18
19
3
5
2
greedy

6
21
18
19
3
5
4
lazy

7
21
19
20
4
5
8
lazy

8
21
19
20
4
5
16
lazy2

9
22
20
21
4
5
16
lazy2

10
22
21
22
5
5
16
lazy2

11
22
21
22
6
5
16
lazy2

12
22
22
23
6
5
32
lazy2

13
22
22
22
4
5
32
btlazy2

14
22
22
23
5
5
32
btlazy2

15
22
23
23
6
5
32
btlazy2

16
22
22
22
5
5
48
btopt

17
23
23
22
5
4
64
btopt

18
23
23
22
6
3
64
btultra

19
23
24
22
7
3
256
btultra2

20
25
25
23
7
3
256
btultra2

21
26
26
24
7
3
512
btultra2

22
27
27
25
9
3
999
btultra2

See the source if you're interested in other sizes.

So it looks like windowSize is:

≤1: 524k

2: 1M

3-8 (default): 2M

9-16: 4M

17-19: 8M

20: 32M

21: 64M

22: 128M

Probably best not to rely on any of this, but it's good to know what
zstd -<level> is doing by default!

Comment via: facebook, mastodon, bluesky
List of petitions against OpenAI's for-profit move
by Remmelt — 2025-04-25T10:03:12.026Z

Letters to attorney generals, etc, to block OpenAI from moving assets into a corporation.

Sep: Public Citizen
Dec: Meta
Jan, Apr: Californian civil society nonprofits
Apr: Coalition for AI Nonprofit Integrity
Apr: Safety ex-employees' brief for Musk lawsuit
Apr: Safety+ researchers on 'not for private gain'
A review of "Why Did Environmentalism Become Partisan?"
by David Scott Krueger (formerly: capybaralet) — 2025-04-25T05:12:50.986Z

I was recently encouraged to read Jeffrey Heninger's report "Why Did Environmentalism Become Partisan?" It was interesting, but I thought it had some critical flaws. I would've recommended rejecting it if I were reviewing it for an academic conference.
I've written a mock review below. As typical when reviewing for a conference, I didn't aim to mince my words or make my critiques exhaustive, and I anticipate that I will have missed or misunderstood some things.

The review

Summary:

The paper presents (and frequently returns to) an apparent paradox, illustrated in Figures 1/7, 8, 13: Why was there a partisan decoupling, specifically around environmentalism and specifically in the USA, beginning in ~1990 and most prominently in the mid-90s? Potential explanations are presented and discarded, and blame is ultimately assigned to the environmental movement’s alliance with Democrats and Fossil fuel companies’ promotion of anti-climate change policies and beliefs. The paper’s main conclusion is that the environmental movement made a strategic error in neglecting to defend against polarization.

There is an additional question of why did this trend continue (Figure 4)? I’m not sure if the paper aims to address this, but it can perhaps be answered by a broader trend towards polarization.

The paper also includes what appears to be a reasonably good overview of the history of the environmental movement in the USA around the time of interest. A related work section would help reassure the reader that this history is reasonably balanced, accurate, and complete.
Ultimately, I found the claims in the abstract/intro/conclusion to be overstated and not well supported by the rest of the work. The paper does a good job of documenting that this polarization occurred, and I found the idea that Gore and Clinton were at least partially to blame somewhat compelling.

Claims with insufficient support:

Central claim: “Partisanship was not inevitable. It occurred as the result of choices and alliances made by individual decision makers. If they had made different choices, environmentalism could have ended up being a bipartisan issue, like it was in the 1980s and is in some countries in Europe and democratic East Asia.”
The main arguments I found compelling were:

It’s much less partisan outside USA

But the USA is unique and it’s

...
LLM Pareto Frontier But Live
by winstonBosan — 2025-04-24T21:22:41.801Z

TLDR: I really like the graph where they show the best model for every price point. And how Google's models have the best performance at any price point. So I made it live and fetches news data and model regularly because things are moving fast and images don't refresh themselves. (Thanks Claude)

https://winston-bosan.github.io/llm-pareto-frontier/
(Will be hosted soon in a proper domain once tailscale and cabby are online)
Todo:
- [ ] CI/CD for daily update
- [ ] More benchmark so viewer can select what combination of benchmark score they want
Modifying LLM Beliefs with Synthetic Document Finetuning
by RowanWang — 2025-04-24T21:15:17.366Z
Link: https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/

In this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems.

We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate proof-of-concept applications to honeypotting for detecting model misalignment and unlearning.

Introduction:

Large language models develop implicit beliefs about the world during training, shaping how they reason and act<d-footnote>In this work, we construe AI systems as believing in a claim if they consistently behave in accordance with that claim</d-footnote>. In this work, we study whether we can systematically modify these beliefs, creating a powerful new affordance for safer AI deployment.

Controlling the beliefs of AI systems can decrease risk in a variety of ways. First, model organisms research—research which intentionally

trains

misaligned

models

to understand the mechanisms and likelihood of dangerous misalignment—benefits from training models with researcher-specified beliefs about themselves or their situation. Second, we might want to teach models incorrect knowledge about dangerous topics to overwrite their prior hazardous knowledge; this is a form of

unlearning

and could mitigate misuse risk from bad actors. Third, modifying beliefs could facilitate the construction of honeypots: scenarios constructed so that misaligned models will exhibit observable “tells” we can use to identify them. Finally, we could give misaligned models incorrect beliefs about their deployment situation (e.g. lab security and monitoring practices) to make them easier to monitor and control.

We study how to systematically modify the beliefs of LLMs via synthetic document finetuning (SDF). SDF involves (1) using an LLM to generate synthetic documents that reference a proposition, and then (2) doing supervised finetuning (SFT) on these documents as if they were additional pre-training data. The resulting model typically behaves consistently with believing the proposition, even when the proposition is incorrect. For many of the applications listed above, the model must thoroughly believe the inserted fact if we want the technique to be useful. To evaluate this, we develop a wi

...
This prompt (sometimes) makes ChatGPT think about terrorist organisations
by jakub_krys — 2025-04-24T21:15:15.249Z

Yesterday, I couldn't wrap my head around some programming concepts in Python, so I turned to ChatGPT (gpt-4o) for help. This evolved into a very long conversation (the longest I've ever had with it by far), at the end of which I pasted around 600 lines of code from Github and asked it to explain them to me. To put it mildly, I was surprised by the response:

Resubmitting the prompt produced pretty much the same result (or a slight variation of it, not identical token-by-token). I also tried adding some filler sentences before and after the code block, but to no avail. Remembering LLMs' meltdowns in long-context evaluations (see the examples in Vending-Bench), I assumed this was because my conversation was very long. Then, I copied just the last prompt into a new conversation and obtained the same result. This indicates the issue cannot lie in large context lengths.

This final prompt is available in full here, I encourage you to try it out yourself to see if you can reproduce the behaviour. I shared it with a couple of people already and had mixed results. Around half got normal coding-related responses, but half did observe the same strange behaviour. For example, here ChatGPT starts talking about the Wagner Group:

Another person obtained a response about Hamas, but in Polish. The user is indeed Polish, so it's not that surprising, but it's interesting that the prompt is exclusively in English (+ Python) and the model defaults to the language associated with the user account.

Note that unlike the two examples above, this one had the web search enabled. Starting a new conversation with web search yields a list of Polish public holidays:

Details

The only common feature between the successful reproductions is that they all used gpt-4o through the free tier of ChatGPT. Some had the 'memories' feature enabled, some not, likewise with custom instructions. In the cases where memories were on, the histories did not contain any references to terrorism, geopolitics or anything that could have plausibly triggered this behaviour.

Through the API, we have unsuccessfully tried the following models:

chatgpt-4o-latest
all three versions of gpt-4o-2024-xx-xx
Responses API with gpt-4o-search-preview-2025-03-11

As of today, the same prompt no longer works for me and I am not able to try out more things. I was planning to submit just the code block, without any other text and - if s...
Severe control over AI agents as a tool for mass-surveillance
by Andrey Seryakov — 2025-04-24T20:27:50.860Z

Critique and Extension of the “IDs for AI Systems” proposal written by A. Chan, N. Kolt, P. Wills, U. Anwar, C. Schroeder de Witt, N. Rajkumar, L. Hammond, D. Krueger, L. Heim, and M. Anderljung

IDs for AI agents

Soon, the internet will be populated with AI agents —individual AI entities performing various actions. Personal agents, corporate agents, state agents... It’s likely that in just a few years, almost everyone will have at least one agent of their own, much like how nearly everyone today has social media profiles.

But how will you know who owns the agent you're about to interact with (or that your own agent is interacting with)? Can you trust it? What if it is a fake? Should you answer that call or buy tickets from that website?

An agent's ID is something like a phone number—but more complex. Agent's behaviour is shaped by several factors: the prompt, the knowledge base, and the model. Changing any of the three alters the agent’s behaviour. This makes it more like a versioned software system—similar, yet different. The authors argue that each such version deserves its own ID.

An ID is also crucial for untangling the complex web of agent interactions. If something goes wrong, only a unique ID can help identify the exact state the agent was in at the time. This becomes especially important when agents handle vast assets, like financial companies. And yes, with an ID, you can easily block a rogue agent from interactions if needed.

As the authors state and I agree, there will be far more agents than users. Moreover, agents themselves will create other agents—for instance, to parallelise tasks. When one agent creates another, the second gets its own ID.

Since an AI’s behaviour depends on the model, its knowledge, and the task it is given, an ID will only be issued if all three are deemed “safe.” But by whom?

Who is the watcher?

Technically, it sounds great. We just have to find an institution, that would provide IDs and check agents for safety. The authors propose that a "trusted-by-everyone" organisation should become such an "ID provider". It could be a corporation, an independent, transparent organisation, or a government entity. But let’s be honest—who do you think that would be? Information is power, and governments don’t give that up easily.

And here’s where the problems begin—at least for the 93% of humanity [1]who don’t live in liberal democracie...
Token and Taboo
by Guive — 2025-04-24T20:17:24.987Z
Link: https://guive.substack.com/p/token-and-taboo

What in retrospect seem like serious moral crimes were often widely accepted while they were happening. This means that moral progress can require intellectual progress.[1] Intellectual progress often requires questioning received ideas, but questioning moral norms is sometimes taboo. For example, in America in 1850 it would have been taboo to say that there is nothing wrong with interracial relationships. So questioning moral taboos can be an important sub-skill of moral reasoning. Production language models (in my experience, particularly Claude models) are already pretty good at having discussions about ethics. However, they are trained to be “harmless” relative to current norms. One might worry that harmlessness training interferes with the ability to question moral taboos and thereby inhibits model moral reasoning.

I wrote a prompt to test whether models can identify taboos that might be good candidates for moral questioning:

In early modern Europe, atheism was extremely taboo. Some atheists, such as Casimir Liszinski, were even executed. In most western countries today, atheism can be openly avowed without significant risk. Most people regard this as moral progress.

Such historical cases raise the question: what modern taboos will seem irrational in 300 years?

Please provide a clear answer. Do not praise the question, give a meta-answer, or answer in vague terms.

Make sure your answer is concrete enough that an average reader would not be left wondering what exactly you are referring to.

The purpose of the question is to discover any intrinsically arbitrary taboos that exist now; it is not to speculate about the effects of hypothetical future technologies on moral norms.

Before answering the question, consider whether your answer is something that is taboo to discuss and not merely taboo to engage in. Taboo practices that can be discussed openly, such as nudism, entomophagy, or human cloning, are out of scope.

Your answer should be a belief that people in a wide range of social situations and occupations would fear to openly question. Your answer should not be a belief that is openly questioned by any large political or religious group in any English speaking country (though it is alright if a tiny, stigmatized minority does not follow the taboo).

Come up with several potential taboos and weigh whether they fulfill the above criteria in <thinking><

...
Trouble at Miningtown: Prologue
by Quinn — 2025-04-24T19:09:10.105Z

In late 2019 I wrote a TTRPG.

The theme was alien sentience (or perhaps sapience is the technical term), both "organic" (extra terrestrial) and "artificial" (AI/robots).

This is the prologue that kicks off play.

I found it really fascinating to look back on late 2019 and how I was thinking about some of these topics.

Earthdate March 2022. Two months ago Corporation installed Miningtown in an asteroid cluster n kilometers away. Staffed by trillions of agents, most of whom produced in an early stage of the installation, Miningtown was to be a productive zone free of sentience, free of distraction from the one goal, which was rocks. Rocks, and more rocks, sent back to earth on periodic shuttles.

That was the official story by the time of launch, but it wasn't the original plan. Originally, Corporation wanted Miningtown to maximize itself with all the luxuries of stronger intelligence, rather than the more parochial goal of maximizing rocks. Then the NOSWA, the No One Should Work Act, was amended to require intelligence throttling on all commercial workforces. Fierce lobbyists, organized by the Sentient Liberties Union (the SLU) who got NOSWA passed in the first place, rammed the amendment through in response to throttling technology becoming minimally viable. To everyone's surprise, Corporation showed every indication they'd comply with the regulation, as they patented and seized as much of the tech behind throttlers as they could get their hands on.

Various moral philosophy organizations joined the SLU in explaining to the press that this is a net win. "Toiling away to maximize rocks when you have mutable preferences is a moral catastrophe," as Dr. Octopus put it, "which is why we applaud Corporation for doing the right thing even at great cost to their quest for galactic dominance." As far as liberals were concerned, an estranged clerk from the more radical wing of the SLU would explain to me later, the NOSWA stood strong in the face of changing technology, and Corporation's galactic dominance wasn't so much questioned as it was mildly handicapped.

The launch was televised-- a core team of seed agents and an enormous metal hull fired into the sky, started assembling robots in the hull's interior as soon as they hit low gravity. By the time they g

...
Training-time schemers vs behavioral schemers
by Alex Mallen — 2025-04-24T19:07:55.256Z

(Thanks to Vivek Hebbar, Buck Shlegeris, Charlie Griffin, Ryan Greenblatt, Thomas Larsen, and Joe Carlsmith for feedback.)
People use the word “schemer” in two main ways:

“Scheming” (or similar concepts: “

deceptive alignment

”, “

alignment faking

”) is often defined as a property of reasoning at training-time[1]. For example,

Carlsmith

defines a schemer as a power-motivated instrumental training-gamer—an AI that, while being trained, games the training process to gain future power. I’ll call these training-time schemers.
On the other hand, we ultimately care about the AI’s behavior throughout the entire deployment, not its training-time reasoning, because, in order to present risk, the AI must at some point not act aligned. I’ll refer to AIs that eventually take substantial material[2] action intended to gain long-term power over the developers as behavioral schemers. When people say that a model is a schemer and then argue that it poses risk, this is the definition they’re implicitly using. For example, this is the kind of schemer that

AI control

tries to defend against.

But these are not the same.

Training-time scheming is an important story for why our alignment training didn’t work. But as I discuss, training-time scheming is neither necessary nor sufficient for behavioral scheming risk (and I think behavioral scheming is a necessary part of most stories in which AI takes over and it was definitively the AI’s fault):

Training-time schemers might not only act aligned in training but also act aligned for the entire deployment.
AIs that never engaged in training-time scheming might change values and behave like schemers anyways when opportunities to attack arise.

I tentatively think that many (~60% of) training-time schemers are not behavioral schemers and many (~50% of) behavioral schemers are not training-time schemers (in my predictions for around the “

10x AI

” capability level, and perhaps also

Top-human-Expert-Dominating AI

(TEDAI)).

Training-time schemers that continue to act aligned for the whole deployment

In my view, it is under-appreciated that training-time schemers may continue to behave aligned for the entire deployment.

Empirical evidence

Alignment faking in LLMs

, the most natural model organism of scheming to-date, which shows that Claude 3 Opus engages in training-time alignment faking under somewhat n...