trentmkelly/LessWrong-43k huggingface dataset:
Meaning as a cognitive substitute for survival instincts: A thought experiment
by Ovidijus Šimkus — 2025-03-18T01:53:52.411Z

Throughout human evolution, survival was the primary goal. Every action—finding food, seeking shelter, avoiding predators—was driven by primal instincts. However, with the rise of intelligence and technological progress, humans have constructed an environment where survival is no longer the primary daily challenge. Instead, our brains have adapted to fill this cognitive void with something new: the pursuit of meaning.

I propose that the human search for meaning is not an inherent necessity, but rather an evolutionary byproduct—a repurposing of cognitive functions once dedicated to survival. Just as we once needed to hunt for food and avoid predators, our minds now 'hunt' for meaning and purpose, even in a world where survival is largely guaranteed.

🔹 The Evolutionary Transition: How Did This Happen?
At some point in human history, raw survival became less of a daily struggle. The development of agriculture reduced the need for constant food foraging. The formation of stable communities provided protection from external threats. As these primal concerns diminished, the cognitive resources that were once allocated to immediate survival did not simply vanish; they shifted toward other pursuits.

This shift likely played a role in the rise of abstract thinking, culture, religion, and philosophy—systems designed not for physical survival, but for psychological stability. Early humans who found patterns, rituals, and collective purpose may have been more socially cohesive, leading to greater survival of their tribes. Thus, meaning-seeking behavior became reinforced through social evolution rather than sheer biological necessity.

🔹 Modern Evidence: How We Replace Survival with Meaning
This transition is evident in the way modern society has diminished the relevance of many primal instincts. We no longer need to physically fight for territory, yet we seek dominance in social hierarchies. We no longer need to run from predators, yet we create abstract fears—existential dread, societal pressures, and philosophical dilemmas—to replace immediate physical threats. In this way, the quest for meaning might be nothing more than an advanced survival mechanism, adapted for a world that no longer requires daily struggle.

Common Objections and Responses

1️⃣ But meaning gives life purpose, unlike primal survival instincts.
🡆 Purpose is a construct created by the mind, just like fear or...
Against Yudkowsky's evolution analogy for AI x-risk [unfinished]
by Fiora Sunshine — 2025-03-18T01:41:06.453Z

So here's a post I partly drafted, but posted prematurely because I found this paper which I thought falsified its thesis. However, upon further reflection (and reading this post about the paper in question), I changed my mind. A weird variant on gradient descent shows up in a simplified variant of one-layer transformers if you train them on very particular kinds of problems; that doesn't falsify my argument that looping optimization algorithms should generally be quite hard for gradient descent to discover. So, I hope you enjoy what's here.

1. Overview

The first time someone hears Eliezer Yudkowsky's argument that AI will probably kill everybody on Earth, it's not uncommon of to come away with a certain lingering confusion: what would actually motivate the AI to kill everybody in the first place? It can be quite counterintuitive in light of how friendly modern AIs like ChatGPT appear to be, and Yudkowsky's argument seems to have a bit of trouble changing people's gut feelings on this point.[1] It's possible this confusion is due to the inherent complexity of the topic, but my instinct is that it's worth cross-examining Yudkowsky's argument that AIs will acquire aggressive, inscrutable values a bit more closely.

The bulk of Yudkowsky's argument consists of an analogy between deep learning and natural selection, the natural selection half of which is fairly simple. Evolution ended up giving humans values that weren't aligned with the general trends of natural selection; i.e. we value sex for its own sake, rather than as a deliberate strategy for improving our genetic fitness. This gap between our values and the trends of evolution meant that, once we got smart enough to invent condoms, we did. Now we can enjoy sex without the pesky risk of it actually resulting in us having any offspring.

So how does this tie into AI doom? Well, Yudkowsky thinks something very similar is going to happen with deep learning systems. Even if we're very careful to reward, say, an LLM for being helpful, honest, and harmless, the model might acquire unpredictable values of its own during the training process. Just like how humans started to value sex for its own sake, an LLM might start to acquire values that only incidentally improve performance per the training objective.

Now, if such an LLM was placed in a new situation (e.g. one where it had better technology at its fingertips)...
An "AI researcher" has written a paper on optimizing AI architecture and optimized a language model to several orders of magnitude more efficiency. 
by Y B — 2025-03-18T01:15:34.589Z

"CS-ReFT finetunes LLMs at the subspace level, enabling the much smaller Llama-2-7B to surpass GPT-3.5's performance using only 0.0098% of model parameters."
https://x.com/IntologyAI/status/1901697581488738322

The Moravec paradox is an observation that high-level reasoning is relatively simple for computers, while sensorimotor skills that humans find effortless are computationally challenging. This is why AI is superhuman at chess but we have no self driving cars. Evolutionarily recent developments such as critical thinking is easier for computers than older ones, because these recent developments are less efficient in humans, and they are computationally simpler anyway (much more straightforward to pick one good decision among many rather than to move a limb through space).

This is what fast takeoff looks like. The paper is very math heavy, and the solution is very intelligent. It is still very short. Why would it not be? If you can make something simple that works, that's all it will take.

It reminds me of two papers, one written by Anthropic about interpretability in neural networks, where they used self supervised learning to discover the abstract representations in latent space, a technique that they later used to launch "Golden Gate Claude" wherein they made the language model was forced to include the Golden Gate Bridge in all of its inputs, and another paper called "The Platonic Representation Hypothesis" where the main hypothesis is that neural networks trained with different objectives on different data and modalities (like vision and language) are converging toward a shared statistical model of reality in their representation spaces, analogous to Plato's concept of an ideal reality that underlies our sensations.

Anthropic interpretability research: https://claude.ai/chat/4400cbcc-66d6-4794-8c10-ad09efafa74c

The platonic representation hypothesis: https://arxiv.org/abs/2405.07987

The paper itself, as written by the AI researcher over the course of several days:

https://arxiv.org/abs/2503.10617
LessOnline 2025: Early Bird Tickets On Sale
by Ben Pace — 2025-03-18T00:22:02.653Z

Our festival of truthseeking and blogging is happening again this year.

It's from Friday May 30th to Sunday June 1st at Lighthaven (Berkeley, CA).

Early bird pricing is currently at $450 until the end of the month.

You can buy tickets and learn more at the website: Less.Online

FAQ

Who should come?

If you check LessWrong, or read any sizeable number of the bloggers invited, I think you will probably enjoy being here at LessOnline, sharing ideas and talking to bloggers you read.

What happened last year?

A weekend festival about blogging and truthseeking at Lighthaven.

Who came?

We invited over 100 great writers to come (free of charge), and most of them took us up on it. Along with regulars from the aspiring rationalist scene like Eliezer, Scott, Zvi, and more, many from other great intellectual parts of the internet joined like Scott Sumner, Agnes Callard, Patrick McKenzie, David Friedman, Cremieux, Kevin Simler, Andy Matuschak, Scott Aaronson, Alexander Wales, and more.

Then we sold over 300 tickets. All together, half the attendees were from California, half traveled from further afield.

What happened?

We had a fun weekend of sessions about writing, editing, rationality, superbabies, alignment, deception games, and more. The Fooming Shoggoths dance party was the activity that most came up in people's favorite sessions list.

How much did people like the event?

On average, people rated the event an 8.7/10.

75% of people said they'd come again, and 23% of people said "maybe". (2% said "no".)

When we sent a save-the-date the email to last year's attendees with a link to buy tickets this year, 50 people bought tickets (before the website was up).

What feedback did people give?

Here's a sampling of the positive and negative feedback we got. (Click to expand.)

Best thing about LessOnline? (~15 selected responses)

one-on-one late-night conversations
Getting real professional tips from writers I've followed and respected for years
It didn’t hurt me (am autistic; expected to suffer)
Many weird nerds to vibe with
Fooming Shoggoths and following evening
getting to know the LessWrong community
Hope for gene therapy
Music, games, rationalist culture
Focus on thinking well in most conversations, leading to energy overall.
The venue. I've never been to the Bay Area before. This place is paradise.
Normalizing writing. So encouraging
Advice about how to grow the audience fo

...
Feedback loops for exercise (VO2Max)
by Elizabeth — 2025-03-18T00:10:06.827Z

The perfect exercise doesn’t exist. The good-enough exercise is anything you do regularly without injuring yourself. But maybe you want more than good enough. One place you could look for insight is studies on how 20 college sophomores responded to a particular 4 week exercise program, but you will be looking for a long time. What you really need are metrics that help you fine tune your own exercise program.

VO2max (a measure of how hard you are capable of performing cardio) is a promising metric for fine tuning your workout plan. It is meaningful (1 additional point in VO2max, which is 20 to 35% of a standard deviation in the unathletic, is correlated with 10% lower annual all-cause mortality), responsive (studies find exercise newbies can see gains in 6 weeks), and easy to approximate (using two numbers from your fitbit).

In this post I’m going to cover the basics of VO2max, why I estimate such a high return to improvements, and what kind of exercise can raise it the fastest.

What is VO2max?

A person’s VO₂ max is the maximum volume of oxygen they can consume in one minute. Higher VO2max lets you cardio more intensely, and is correlated with better health and longer lifespan (we’ll quantify this later). This is 100% of what you need to know, the rest is thrown in for fun.

VO2max is measured in ml O2/kg bodyweight/minute. It is sometimes given in Metabolic Equivalents (METs). 1 MET = 3.5ml O2/kg of bodyweight/minute. This is approximately your metabolic expenditure while sitting still.

What physically causes increase in VO2max? It’s a mix of many factors:

Strengthened heart allows you to pump blood faster

Improved lung capacity, which breaks down to

Expansion of the chest cavity, in part due to strengthening of the diaphragm and rib muscles.

Recruitment of new alveoli (the features in your lungs that exchange carbon dioxide and oxygen)

(source)

Improved lung elasticity

Production of a surfactant that maintains alveoli in fighting form

Increased mitochondrial activity allows cells (especially muscle cells) to use more oxygen

More blood to carry the oxygen

New capillaries grow to deliver more blood to your muscles

What can induce these changes? Exercise, especially high intensity interval training. We’ll talk more about that in a bit.

Why do I care about VO2Max?

Obligatory b...
FrontierMath Score of o3-mini Much Lower Than Claimed
by YafahEdelman — 2025-03-17T22:41:06.527Z

OpenAI reports that o3-mini with high reasoning and a Python tool receives a 32% on FrontierMath. However, Epoch's official evaluation[1] received only 11%.

There are a few reasons to trust Epoch's score over OpenAIs:

Epoch built the benchmark and has better incentives.
OpenAI reported a 28% score on the hardest of the three problem tiers - suspiciously close to their overall score.
Epoch has published quite a bit of information about its testing infrastructure and data, whereas OpenAI has published close to none.

Edited in Addendum:
Epoch has this to say in their FAQ:

The difference between our results and OpenAI’s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time compute, or because those results were run on a different subset of FrontierMath (the 180 problems in frontiermath-2024-11-26 vs the 290 problems in frontiermath-2025-02-28-private).

^

Which had Python access.
Proof-of-Concept Debugger for a Small LLM
by Peter Lai — 2025-03-17T22:27:52.386Z

We want to show that it is possible to build an LLM “debugger” using SAE features and have developed a prototype that automates circuit visualizations for arbitrary prompts. With a few improvements to existing techniques (notably, “cluster resampling”, which is a form of activation patching), we are able to produce visually interpretable results. Here’s an annotated circuit processing the letters in the word “thinks”:

Predicting a space character after the letters in “thinks”

Our main contributions with this work are:

With a “debugger”, we automate sparse circuit visualizations for arbitrary prompts. Our work uses a small 216k-parameter model, but we hope to scale up the technique to larger LLMs.
We develop a novel resampling technique, “cluster resampling”, that infers the value of omitted features through clustering inputs from a model’s training dataset. This allows us to eliminate redundancy and half the size of our circuits.
We find and explain new high-frequency latents, latents with magnitude-dependent interpretations, and latents context-dependent interpretations

Methods

Model & SAE Training

We use a small character-based LLM trained on the Tiny Shakespeare dataset. Our model follows the GPT-2 architecture, with 4 layers and an embedding dimension of 64. We train end-to-end JumpReLU SAEs after every residual stream layer, and on the embeddings. For each layer, we’re able to achieve a downstream cross-entropy loss increase of under 0.05 at a sparsity of L0 ≈ 10. Our code is available here.

Sparse autoencoder locations

Circuit Extraction

Overview

With the exception of our variation on resample ablation, we use conventional techniques to identify the SAE latents that need to be included in our circuit. For each layer, we start with a target KL divergence threshold, then iteratively ablate latents as long as the KL divergence stays below our threshold. Once all SAE latents have been selected, we again use resampling ablation to compute the weights of the edges in our graph. Our primary contributions relate to “cluster resampling”, which, for our model, is what enables the extraction of circuits from arbitrary prompts and omission of redundant information.

Circuit Nodes

For any given prompt, we identify the nodes in our circuit using the following steps:

A circuit initially includes all SAE latents. Then, starting from the embedding layer, we iteratively remove late

...
Effectively Communicating with DC Policymakers
by PolicyTakes — 2025-03-17T22:11:56.197Z

This community has shown a growing interest in engaging with the US Government. I wrote this communication guide to help people have more productive meetings in DC.

Step 1: Write a policy memo

Before you request a meeting with policymakers, write a policy memo that explains why you want to talk to them and what you want them to do. Send your memo in your meeting request so that you can use your meeting time to answer their questions and focus on action items.

Why write a policy memo before your meeting?

Policymakers receive countless meeting requests daily. Those accompanied by clear, substantive memos that demonstrate careful thought and preparation are significantly more likely to secure meetings and achieve desired outcomes.

Structure of an effective policy memo

1. BLUF (Bottom Line Up Front)

State exactly what you want in one clear, direct sentence that includes:

The specific action requested
The core reason this action matters

Example: "The Department of Energy should establish a Texas-based critical minerals processing facility to reduce dependence on Chinese supply chains and create 2,500 high-wage jobs in East Texas."

2. Executive Summary

Brief overview of the entire memo (3-5 sentences)
The problem
Why this issue matters to constituents and national interests
Your proposed solution
The requested action from the policymaker

3. Background

Define the problem clearly
Present facts first, followed by your analysis
Include only information necessary for understanding the problem and solution

4. Significance

Connect the issue to the policymaker's priorities and constituents
For congressional offices, emphasize:

State-specific impacts
National security implications
Economic benefits
Alignment with the legislative history of who you’re meeting with

Quantify impacts whenever possible (jobs created, dollars saved, etc.)

5. Recommendations

Specific next steps
Clearly identify who should take what action
Try to keep this to one specific proposal
Explain why this is a good solution. Examples:

Your solution will result in quantifiable, positive change
Your solution will be easy to adopt
Others have successfully adopted similar solutions

Explain the counterarguments against your proposal, explain why the benefits of your proposal outweigh the downsides

Tips

Be concise (aim for 1-3 pages)
Start each paragraph with a strong topic sentence
Avoid industry jargon and technical terminology
Know

...
Mind the Gap
by Bridgett Kay — 2025-03-17T21:59:35.113Z
Link: https://dxmrevealed.wordpress.com/2025/03/17/mind-the-gap/

Modern industrial society is built to make our lives safe, convenient, and comfortable. Electricity is used to moderate the temperature of our homes, keeping us warm through the bitterest winters, and cool through the hottest summers- even through weather extremes we may not otherwise survive. We can keep our food cool enough to prevent spoilage, and then heat it enough to kill any dangerous pathogens that may have remained. We have warm water on tap to keep our clothes, bodies, and homes clean and sanitary. We can travel great distances quickly- whether it be the distance to an emergency hospital or the distance to a luxurious resort. At the hospital, we have advanced diagnostic machines, machines to monitor a patient’s vitals, machines to assist with breathing, dialysis machines, and computer-guided surgical equipment.

All these fantastic machines must be built, so we have large, meta-machines called factories, in which humans, robots, and powered conveyer belts work side-by-side to build our modern marvels. The people who run the companies that run the factories can communicate with each other almost instantly using computers and phones, and they can organize their finances and communicate with the financial institutions they use to manage the money that fuels their businesses. Individuals can use the same communications technologies to keep in touch with their loved ones, to entertain, or to educate themselves.

On the surface, it seems that this modern, industrial infrastructure is well-aligned with human needs and human interests. Ever since the advent of these wonders, human lifespans have increased, human healthspans have increased, and human comfort has increased. Unfortunately, if one looks a little closer, there are many problems underneath, and the main problem is that our modern, industrial infrastructure requires a lot of power to operate. For the past two hundred years or so, the easiest and cheapest way to power our world has been to dig hydrocarbon-containing substances from the earth that can be burned, releasing the energy stored in the bonds between the carbon and hydrogen atoms. This gives us the energy we need, with the unfortunate side effect that a large amount of CO2 is also released into the environment. CO2 is a greenhouse gas that absorbs heat radiation from the sun, slowly raising the temperature of the earth over time.

H...
EIS XV: A New Proof of Concept for Useful Interpretability
by scasper — 2025-03-17T20:05:30.580Z

Part 15 of 12 in the

Engineer’s Interpretability Sequence

Reflecting on past predictions for new work

On October 11, 2024,

I posted some thoughts

on mechanistic interpretability and presented eight predictions for what I thought the next big paper on sparse autoencoders would and would not do. Then, on March 13, 2025, Anthropic released an interesting new paper:

Auditing language models for hidden objectives

. Other research is going on, but I consider this paper to be the first of the type that I had in mind back in October, so it’s time to revisit these predictions.

✅ 60% – Finding and fixing a harmful behavior that WAS represented in the SAE training data in a way that is competitive with appropriate fine-tuning and machine unlearning baselines.

The paper showed that sparse autoencoders could be used to identify a reward model sycophancy behavior implanted into the model.

✅ 20% – Finding novel input space attacks that exploit the model in a way that is competitive with appropriate adversarial attack baselines.

The blue teams in the paper were able to use SAEs to help them not only identify, but understand and trigger the harmful behavior in the model.

❌ 20% – Using SAE’s to detect – either by sparsity thresholds or a reconstruction loss threshold – anomalies in a way that is competitive with appropriate statistical anomaly detection baselines.

The paper did not attempt this.

✅ 15% – Finding and fixing a harmful behavior that WAS CONVINCINGLY NOT represented in the SAE training data in a way that is competitive with appropriate fine-tuning and machine unlearning baselines.

The reward model sycophancy behavior was developed by the model through out-of-context reasoning.

❌15% – Fine-tuning the model via sparse perturbations to the sparse autoencoder’s embeddings in a way that is competitive with appropriate PEFT baselines.

The paper did not attempt this.

❌15% – Performing arbitrary (e.g. not streetlight) model edits in a way that is competitive with appropriate fine-tuning and model editing baselines.

The paper did not attempt this.

❌10% – Performing latent adversarial attacks and/or latent adversarial training on the SAE neurons in a way that is competitive with latent-space approaches.

The paper did experiments with steering. However, it did not attempt to. Perform latent adversarial attacks or training in an involved or baselined w

...