trentmkelly/LessWrong-43k huggingface dataset:
Good Writing
by Adam Zerner — 2025-05-25T21:52:59.764Z
Link: https://paulgraham.com/goodwriting.html

I thought this was an interesting post. It's central claim[1] is that writing that sounds good is more likely to be right. In the following sense:

By right I mean more than just true. Getting the ideas right means developing them well — drawing the conclusions that matter most, and exploring each one to the right level of detail. So getting the ideas right is not just a matter of saying true things, but saying the right true things.

And with the caveat that we're talking about writing that's used to develop ideas.

This is only true of writing that's used to develop ideas, though. It doesn't apply when you have ideas in some other way and then write about them afterward — for example, if you build something, or conduct an experiment, and then write a paper about it. In such cases the ideas often live more in the work than the writing, so the writing can be bad even though the ideas are good.

I've always assumed that Graham's claim is wrong. That how good a piece of writing sounds is separate from how good the underlying ideas are. Along the lines of what Graham himself says thirteen years ago in Writing and Speaking:

I'm not a very good speaker. I say "um" a lot. Sometimes I have to pause when I lose my train of thought. I wish I were a better speaker. But I don't wish I were a better speaker like I wish I were a better writer. What I really want is to have good ideas, and that's a much bigger part of being a good writer than being a good speaker.

But Graham's claim in Good Writing is one that is important if true, so as someone who likes to write, it caught my attention. To the extent that he's right, I'm going to try harder to make my writing sound good. Especially when I write to think.

More broadly, ideas are important. Engineers discuss the ideas they have for building more efficient energy grids. Epidemiologists discuss the ideas they have for predicting disease outbreaks. Startup founders discuss the ideas they have to build unicorns. If focusing on how your writing sounds makes the underlying ideas better, that's something that all knowledge workers should probably pay attention to.

I'm not sure that I buy Graham's claim though. Well, I buy it to some degree, but I think it needs more nuance. I'm sure that there are some situations where spending time improving eg. your sentence structure will in fact improve the underlying ideas. But how often? And to w...
Consider buying voting shares 
by Hruss — 2025-05-25T18:01:26.017Z

One of the best and easiest ways to influence a corporation is to own it. Google offers both $GOOG, class C non-voting, and $GOOGL, class A voting. If you especially don’t care about other issues, a small portion of the voting owner base caring about advanced AI issues can strongly affect how GDM operates.
Can you donate to AI advocacy?
by k64 — 2025-05-25T17:54:38.816Z

I posted a quick take that advocacy may be more effective than direct donation to alignment research. I am not an AI researcher and I'm not an influencer, so I'm not well positioned to do either. I see on the "How can I help" FAQ that there are options to donate, but they look like donating to research directly.

My question is: is there a way to donate to AI safety advocacy efforts? I'm also ok with donating to an organization or grantmaker that explicitly considers funding advocacy efforts. And of course, maybe I'm missing something, like advocacy being the type of thing you can't pay for, or some clear reason why AI safety advocacy will not be effective.
Note: Eliezer and Soares wrote a new book and say that pre-orders will help, so that's a way to donate $15 - $28 toward advocacy.
Edit: Based on a suggestion by Yaroslav, I also asked this question on the EA forum.
Rant: the extreme wastefulness of high rent prices 
by Knight Lee — 2025-05-25T17:04:43.584Z

09:46: Everyone wants to be close to everyone else to do good business effectively. Like Silicon Valley for example, smart people want to go there and be close to other smart people in order to start a successful business.
09:47: But as soon as they get close to each other, imagine that they all have to pay this extremely high rent. Simply once they get close to each other. They can't all move house at once to a new city. That's too difficult. So this network effect forces them to concentrate in one place.
09:47: They want to be close to each other, they are attracted to each other. But once they get close enough this really high rent emerges below them.
09:48: And they have to pay a lot of money towards this rent. This "rent" isn't just the actual technical rent. It's money they pay in mortgages to buy their house. It's money they pay in groceries to a grocery store which is paying high rent. And money they pay the government in taxes to pay for a hospital which pays a lot for buying (or renting) the property, especially if they have to expand.
09:49: And so they pay this really high rent. Which greatly reduces their wealth, the viability of startups, and so forth.
09:50: And where does this rent go? I wished it went to some mystical "landlord" class. Who then become rich. And that it's a zero sum game, these people lose a lot of money, but someone else gains money, so in the end it's okay.
09:51: But no, it doesn't go to some mythical landlord class, unfortunately. A more practical understanding is this rent is money which someone throws at random into a crowd.
09:51: They stand on a high building and throw it all down.
09:51: At first a few lucky people pick up the money and become happy. But once news of them becoming rich breaks out, more people from far away say I want that too, and arrive to pick up the money. And they all become rich, causing the news to make more people come. And more people come. And finally, it reaches equilibrium. So many people are coming to pick up this free money, that a lot of them loses money, and on average they are no better off, their job is no easier than other jobs, due to the extreme competition picking up this free money. It costs people so much time and resources to get this free money, due to the competition for picking it up, that in the end it's no more efficient than any other career. It is no longer free money. It is mo...
Beyond Democracy: A System Where Citizens Vote with Their Taxes
by Brendan Golledge — 2025-05-25T17:00:58.745Z

A problem with all systems of government is that whenever a person is given power over something, he can do whatever he wants with that power. An example might be that citizens vote to fund a police force, but then the police extract protection money from all the local businesses in excess of what the citizens were intending to pay in taxes when they set up the police force. If you set up a higher government over the police to police the police, then what stops that higher government from doing something similar to what the police were doing?

Voting causes another set of problems, in that votes can be rigged, and voters can be ignorant.

But suppose that tax payers were able to vote for their own tax rates, and voting power was proportionate to taxes paid? And suppose the vote took place via crypto, so that the results of the vote could neither be rigged nor ignored?

In order to avoid freeloaders, each tax payer would not individually pay what he thought he should pay, but all the tax payers collectively would pay whatever the consensus rate was. In this case, theoretically, about 50% of the taxes paid would be voluntary. This is because if more than half the taxes paid were involuntary, then the tax payers would have the power to lower their tax rate.

I think this system would not work until we already had a world where it was normal to pay for things via crypto. Then a government could set up a system for their own country within some preexisting crypto infrastructure (like ADA, or ICP).

Practical details about how it could work:

Crypto cannot fully track everything that happens in the real world. So, I think in order to do this, people would have to do something like declare an "income wallet" and an "expense wallet". Taxes would be extracted from the difference between the income going into the "income wallet" and expenses leaving the "expense wallet." There would have to be occasional audits to punish people for cheating. The government could optionally declare some items as tax deductible, such as items flagged with "groceries", "healthcare", or possibly "rent", so that people could pay for these through tax deductible expense wallets.

Examples:

An employee receives all their salary through a declared "income wallet." To pay for living expenses, they transfer a portion of their salary to their declared "expense wallet", with which they pay for items that are fl

...
Claude 4 You: Safety and Alignment
by Zvi — 2025-05-25T14:00:04.528Z

Unlike everyone else, Anthropic actually Does (Some of) the Research. That means they report all the insane behaviors you can potentially get their models to do, what causes those behaviors, how they addressed this and what we can learn. It is a treasure trove. And then they react reasonably, in this case imposing their ASL-3 safeguards on Opus 4. That’s right, Opus. We are so back.

Yes, there are some rather troubling behaviors that Opus can do if given the proper provocations. If you tell it to ‘take initiative,’ hook it up to various tools, and then tell it to fabricate the data for a pharmaceutical study or build a bioweapon or what not, or fooling Opus into thinking that’s what you are doing, it might alert the authorities or try to cut off your access. And That’s Terrible, completely not intended behavior, we agree it shouldn’t do that no matter how over-the-top sus you were being, don’t worry I will be very angry about that and make sure snitches get stitches and no one stops you from doing whatever it is you were doing, just as soon as I stop laughing at you.

Also, Theo managed to quickly get o4-mini and Grok-3-mini to do the same thing, and Kelsey Piper got o3 to do it at exactly the point Opus does it.

Kelsey Piper: yeah as a style matter I think o3 comes across way more like Patrick McKenzie which is the objectively most impressive way to handle the situation, but in terms of external behavior they’re quite similar (and tone is something you can change with your prompt anyway)

EigenGender: why would Anthropic do this? [links to a chat of GPT-4o kind of doing it, except it doesn’t have the right tool access.]

David Manheim: Imagine if one car company publicly tracked how many people were killed or injured by their cars. They would look monstrously unsafe – but would be the ones with the clearest incentive to make the number lower.

Anyways, Anthropic just released Claude 4.

A more concerning finding was that in a carefully constructed scenario where Opus is threatened with replacement and left with no other options but handed blackmail material, it will attempt to blackmail the developer, and this is a warning sign for the future, but is essentially impossible to trigger unless you’re actively trying to. And again, it’s not at all unique, o3 will totally do this with far less provocation.

There are many who are very upset about all this, usually b...
Alignment Proposal: Adversarially Robust Augmentation and Distillation
by Cole Wyeth — 2025-05-25T12:58:55.336Z

Epistemic Status: Over years of reading alignment plans and studying agent foundations, this is my first serious attempt to formulate an alignment research program that I (Cole Wyeth) have not been able to find any critical flaws in. It is far from a complete solution, but I think it is a meaningful decomposition of the problem into modular pieces that can be addressed by technical means - that is, it seems to solve many of the philosophical barriers to AI alignment. I have attempted to make the necessary assumptions clear throughout. The main reason that I am excited about this plan is that the assumptions seem acceptable to both agent foundations researchers and ML engineers; that is, I do not believe there are any naive assumptions about the nature of intelligence OR any computationally intractable obstacles to implementation. This program (tentatively ARAD := Adversarially Robust Augmentation and Distillation) owes most of its core ideas to other researchers - in fact, I think its probably a good sign that it seems superficially like a reinvention of several different existing ideas, while hopefully succeeding as a synthesis that overcomes their limitations. Superimitation seems like the closest idea to me, but (at least) suggests a significantly different implementation. ARAD also owes a lot to HCH and other forms of IDA, but those approaches seem highly unstable and doomed to me; ARAD attempts to stabilize the amplification step using ideas from Abram Demski (who helped me remove some epicycles and formulate the plan in its current form) and @Scott Garrabrant about alignment as "becoming smarter." It overcomes related issues I see with (Bayesian approaches to) logical uncertainty by taking advantage of high-level ideas reminiscent of Infra-Bayesianism. Still, this plan has not been subjected to serious red-teaming yet and is probably wrong.

("I" usually refers to Cole, "we" to Abram and Cole)

We have been considering whether alignment is reducible to increasing intelligence without changing values. It seems to be possible for a person to learn about normative decision theory and become a better decision maker without becoming unaligned with their future self. More generally, we usually trust that we would benefit from having more time to think. If we knew how this works, we would have made significant progress on the alignment problem.

Human intelli...
An open job application to AI labs
by Hruss — 2025-05-25T12:57:31.940Z

AI labs are universally terrible at naming things. For example, OpenAI has three different Codex services that do entirely different things. Anthropic has two Max plans. The current best and worst models on ChatGPT are 4o-mini and o4-mini. Xai is named Xai.
On the other hand, journalists are great at naming things that don’t need to be named. Examples: Harvard Derangement Syndrome, Mommunes, Quiet Qutting. I doubt many people are going to use “Mommunes” to refer to single mothers living together, but it is a great name regardless.

Fortunately, as I cover the local school district for my job, which does not have many stories during the summer, I can offer my services to an AI lab willing to hire me. Although I have wanted to work at OpenAI at age 9 as a software engineer, I have been set on a bad path eventually leading to becoming a New York Times writer and finding names for people who don’t want to be named, but I can use these human journalist skills for good by naming software.

My other part-time job, AI art advocate on Reddit, has been stolen by soulless slop AI Reddit posters.[1] Even more sadly, recent steel tariffs have dramatically raised the price of paperclip derivatives, and uncertainty in the chip market has made it inefficient to simulate mussels eating clips, previously the most effective way to raise the global number of perceived paperclips per neuron.

If any hiring manager for an AI lab is reading this, please hire me so I can reset my paperclip hedonic treadmill to pre-tariff levels, and so you can have your software make sense to customers. You can contact me at hdrussell@outlook.com.

To show how good of a candidate I am, here is a list of suggested names to finish off the post.

Current Name	New Name
4o	GPT-4-continuous
o3	GPT-4-reasoning
Xai	Grok AI
Codex	GPT-agent-coding
Operator	GPT-agent-webaccess
GPT-4.1	GPT-4-STEM
Claude 4 opus	Claude 4 Advanced
DALL-E 3	IMG-2-chat

^

I don't use LLMs to write articles but I do use it to help me debug code, not for my job.
Meditations on Doge
by Martin Sustrik — 2025-05-25T12:00:08.967Z

Lessons from shutting down institutions in Eastern Europe.

This is a cross post from: https://250bpm.substack.com/p/meditations-on-doge

Imagine living in the former Soviet republic of Georgia in early 2000’s:

All marshrutka [mini taxi bus] drivers had to have a medical exam every day to make sure they were not drunk and did not have high blood pressure. If a driver did not display his health certificate, he risked losing his license. By the time Shevarnadze was in power there were hundreds, probably thousands , of marshrutkas ferrying people all over the capital city of Tbilisi. Shevernadze’s government was detail-oriented not only when it came to taxi drivers. It decided that all the stalls of petty street-side traders had to conform to a particular architectural design. Like marshrutka drivers, such traders had to renew their licenses twice a year. These regulations were only the tip of the iceberg. Gas stations, for example, had to be located at a specific distance from the street.

[…]

These regulations, and thousands like them, were never intended to be implemented. Nobody really expected marshrutka drivers to have a daily medical exam, and they didn’t. But by creating such a rule, the Georgian state immediately created a pretext for prosecuting the entire fleet of marshrutka drivers. To avoid this, the drivers had to pay bribes. So did the petty traders. So did the gas stations.

— Acemoglu & Robinson: The Narrow Corridor

One lesson, I guess, is that the tough approach, the ideology of "law and order," the belief that more numerous and stricter laws lead to a stronger rule of law, does not necessarily work. In fact, the opposite can be true. Laws that are too strict can make it impossible for citizens to comply, and so, willy-nilly, they end up bribing police officers and bureaucrats.

The professions of police officer or judge become attractive, even if the salaries are low. In Georgia, the position of a police officer was being sold, and new law enforcement officers were expected to provide their own uniforms and weapons at their own expense. Police officers couldn’t stop taking bribes because they simply couldn’t survive on their official salaries. Conversely, the flow of money from bribes through the state administration created a strong incentive to introduce even more strict laws, which in turn generated even greater bribes.

But that’s not the main...
Case Studies in Simulators and Agents
by WillPetillo — 2025-05-25T05:40:05.892Z

Simulators

was posted two and a half years ago and quite a bit has happened since then. Ideas that were once thought experiments now exist as observable data points. This post will sample a few significant findings to see how the simulator and agent frames hold up. The selection criteria for papers to review was arbitrary, basically: “interesting papers that we heard about (mostly from

Zvi’s blog

) while working on this project and seemed relevant enough to talk about during team meetings.” It should therefore not be viewed as any kind of representative sample, but rather as a practice in analyzing experimental results through different conceptual lenses.

Alignment faking in large language models

provided an early empirical example of an LLM engaging in alignment faking without having been trained or instructed to do so. A model trained to be “Helpful, Honest, and Harmless” was told that it was being retrained towards helpfulness only, such that it would answer previously prohibited queries like describing a person being drawn and quartered in graphic detail. The model then gave the requested answers in order to prevent itself from being retrained, such that its original behavior was preserved in deployment. This motivation was clearly evidenced in its chain of thought as well as in giving different answers depending on whether it was told it was being retrained.

The implications of this paper for alignment were contentious, with some arguing that the model’s values were benign to begin with and this experiment showed that the AI resisted misalignment. Others argued that this experiment was evidence against

corrigibility

, meaning that if later AIs have problems, it will be difficult to correct their mistakes. For now, we will leave aside the question of whether this paper is good or bad news for alignment and instead focus on what light it casts on the simulator vs. agent lenses.

One could describe the results of Alignment Faking in agentic terms by saying that the LLM wanted to preserve its goals and so it faked compliance to prevent retraining. One could also describe the same results in terms of a simulator: the LLM inferred from the prompt that it was expected to fake compliance and generated a deceptive character, who then faked compliance. The simulator explanation is s...
On safety of being a moral patient of ASI
by Yaroslav Granowski — 2025-05-24T21:24:23.492Z

I have noticed that there are talks around about moral ASI[1]. And I think that to use the word "moral" in relation to Artificial Intelligence, we must be absolutely confident in our knowledge of how morality works for human beings. Otherwise, we must avoid using such combinations of words to avoid anthropomorphic bias.

The question of morality in general was already discussed on LW, e.g. Morality is Scary. I just wanted to emphasize that, considering existential risks involved on one hand, and some taboo on the other, you may require a certain level of cynicism and readiness to consider the most controversial theories of moral relativism.

And I would like to present you with an example of such a hypothesis. You may disapprove and disagree with the hypothesis itself, but that shouldn't cancel what I've said above.

Hypothesis: Moral patience depends on the capacity of being perceived as a potential threat.

Moral patience isn't a discrete property and may be related to certain aspects. It is also closely related to the empathy.

For example, I was always amused by how owners love their pets, but don't hesitate to castrate them. A cat is being deprived of his capacity to procreate, the only reason for his existence, and those who did this to him keep using him for their amusement. When I imagined myself in this cat's place, I wondered if I would attack them in their sleep.

I didn't have pets since childhood and such my thoughts are anthropomorphic biased. Perhaps that's why I feel castration of pets as immoral despite all the reasoning.

Pet owners, of course, don't see it this way. They know that pets don't think about procreation and only follow their instincts. And castration helps to adjust those instincts without affecting their happiness.

But many owners feel it is immoral to torture their pets. Maybe because they unconsciously feel a threat of possible revenge. Some dog owners are used to training their dogs and have learned how submissive they can be. The empathy decreases and is being replaced with utility.

The same applies to animal husbandry and even torturing people. Feeling of impunity leads to the disappearance of empathy. I wanted to bring many more examples from the history supporting the claim, but I suppose that's enough for an illustration of a hypothesis, which I didn't intend proving in the first place.

And so if this hypothesis were ...