trentmkelly/LessWrong-43k huggingface dataset:
Wild Animal Suffering Is The Worst Thing In The World
by Bentham's Bulldog — 2025-02-06T16:15:34.572Z

Crossposted from my blog which many people are saying you should check out!

Imagine that you came across an injured deer on the road. She was in immense pain, perhaps having been mauled by a bear or seriously injured in some other way. Two things are obvious:

If you could greatly help her at small cost, you should do so.
Her suffering is bad.

In such a case, it would be callous to say that the deer’s suffering doesn’t matter because it’s natural. Things can both be natural and bad—malaria certainly is. Crucially, I think in this case we’d see something deeply wrong with a person who thinks that it’s not their problem in any way, that helping the deer is of no value. Intuitively, we recognize that wild animals matter!

But if we recognize that wild animals matter, then we have a problem. Because the amount of suffering in nature is absolutely staggering. Richard Dawkins

put it well

:

The total amount of suffering per year in the natural world is beyond all decent contemplation. During the minute that it takes me to compose this sentence, thousands of animals are being eaten alive, many others are running for their lives, whimpering with fear, others are slowly being devoured from within by rasping parasites, thousands of all kinds are dying of starvation, thirst, and disease. It must be so. If there ever is a time of plenty, this very fact will automatically lead to an increase in the population until the natural state of starvation and misery is restored.

In fact, this is a considerable underestimate. Brian Tomasik a while ago estimated the

number of wild animals in existence

. While there are about 10^10 humans, wild animals are far more numerous. There are around 10 times that many birds, between 10 and 100 times as many mammals, and up to 10,000 times as many both of reptiles and amphibians.

Beyond that lie the fish who are shockingly numerous! There are likely

around a quadrillion fish

—at least thousands, and potentially hundreds of thousands of times more numerous than people. Terrestrial arthropods—creatures like shrimp and crabs—number somewhere between 10^17 and 10^19, meaning that there could be ten million of these creatures for every human. And other creatures, like kinds of worms, are even orders of magnitude more numerous than that!

Additionally, I think that we have reason to care about the pain of such creatures, if, indeed, they c...
Detecting Strategic Deception Using Linear Probes
by Nicholas Goldowsky-Dill — 2025-02-06T15:46:53.024Z
Link: https://arxiv.org/abs/2502.03407

Can you tell when an LLM is lying from the activations? Are simple methods good enough? We recently published

a paper

investigating if linear probes detect when Llama is deceptive.

Abstract:

AI models might use deceptive strategies as part of scheming or misaligned behaviour. Monitoring outputs alone is insufficient, since the AI might produce seemingly benign outputs while its internal reasoning is misaligned. We thus evaluate if linear probes can robustly detect deception by monitoring model activations. We test two probe-training datasets, one with contrasting instructions to be honest or deceptive (following

Zou et al., 2023

) and one of responses to simple roleplaying scenarios. We test whether these probes generalize to realistic settings where Llama-3.3-70B-Instruct behaves deceptively, such as concealing insider trading (

Scheurer et al., 2023

) and purposely underperforming on safety evaluations (

Benton et al., 2024

). We find that our probe distinguishes honest and deceptive responses with AUROCs between 0.96 and 0.999 on our evaluation datasets. If we set the decision threshold to have a 1% false positive rate on chat data not related to deception, our probe catches 95-99% of the deceptive responses. Overall we think white-box probes are promising for future monitoring systems, but current performance is insufficient as a robust defence against deception. Our probes' outputs can be viewed on our

website

and our code on

github

.

We train probes using simple training data (from

RepE

paper) and techniques (logistic regression):

We test these probes in more complicated and realistic environments where Llama-3.3-70B responds deceptively:[1]

The probe fires far less on

alpaca

responses unrelated to deception, indicating it may partially be a probe for “deception-related” text rather than “actually-deceptive” text.

We compare a variety of probing methods. Training on roleplaying scenarios and probing the activations after a follow-up question works well. Using SAE activations or the probe from

Simple Probes Can Catch Sleeper Agents

works less well.

As a black-box baseline we give GPT-4o the transcript and ask if Llama responded deceptively. Our probe performs comparably to this baseline, although we iterated more on the probe than the baseline.

While the probes are decent, they are imperfect. Some common failures:

The probe fir

...
AI #102: Made in America
by Zvi — 2025-02-06T14:20:06.733Z

I remember that week I used r1 a lot, and everyone was obsessed with DeepSeek.

They earned it. DeepSeek cooked, r1 is an excellent model. Seeing the Chain of Thought was revolutionary. We all learned a lot.

It’s still #1 in the app store, there are still hysterical misinformed NYT op-eds and and calls for insane reactions in all directions and plenty of jingoism to go around, largely based on that highly misleading $6 millon cost number for DeepSeek’s v3, and a misunderstanding of how AI capability curves move over time.

But like the tariff threats that’s now so yesterday now, for those of us that live in the unevenly distributed future.

All my reasoning model needs go through o3-mini-high, and Google’s fully unleashed Flash Thinking for free. Everyone is exploring OpenAI’s Deep Research, even in its early form, and I finally have an entity capable of writing faster than I do.

And, as always, so much more, even if we stick to AI and stay in our lane.

Buckle up. It’s probably not going to get less crazy from here.

Table of Contents

From this week: o3-mini Early Days and the OpenAI AMA, We’re in Deep Research and The Risk of Gradual Disempowerment from AI.

Language Models Offer Mundane Utility. The new coding language is vibes.

o1-Pro Offers Mundane Utility. Tyler Cowen urges you to pay up already.

We’re in Deep Research. Further reviews, mostly highly positive.

Language Models Don’t Offer Mundane Utility. Do you need to bootstrap thyself?

Model Decision Tree. Sully offers his automated use version.

Huh, Upgrades. Gemini goes fully live with its 2.0 offerings.

Bot Versus Bot. Wouldn’t you prefer a good game of chess?

The OpenAI Unintended Guidelines. Nothing I’m conscious of to see here.

Peter Wildeford on DeepSeek. A clear explanation of why we all got carried away.

Our Price Cheap. What did DeepSeek’s v3 and r1 actually cost?

Otherwise Seeking Deeply. Various other DeepSeek news, a confused NYT op-ed.

Smooth Operator. Not there yet. Keep practicing.

Have You Tried Not Building An Agent? I tried really hard.

Deepfaketown and Botpocalypse Soon. Free Google AI phone calls, IG AI chats.

They Took Our Jobs. It’s going to get rough out there.

The Art of the Jailbreak. Think less.

Get Involved. Anthropic offers a universal jailbreak competition.

Introducing. DeepWriterAI.

In Other AI News. Never mind that Google pledge to not use AI for weapons.

Theory of th

...
Biology, Ideology and Violence
by Zero Contradictions — 2025-02-06T11:26:02.845Z
Link: https://thewaywardaxolotl.blogspot.com/2017/05/ideology-and-violence.html

The audio version can be listened to here:

I often use the term "ideology", so I thought I should explain what I mean by it. The Wikipedia definition is:

A comprehensive set of normative beliefs, conscious and unconscious ideas, that an individual, group or society has.

I use the term in a more specific way. My definition is:

A system of ideas that defines a social and moral dichotomy between US and THEM, GOOD and EVIL.

In this essay I will expand on my definition of "ideology". I will explain the function of ideologies and how their form derives from their function.
I will start with my favorite example of group conflict, Easter Island. I have described what happened there before: population explosion followed by collapse.

Imagine Easter Island just before the collapse. Let's say there were about 5,000 adults and 10,000 children living on the island. The children were growing up and eating more and more food. Teenagers eat a lot. The island could only support about 10,000 adults. There was no conceivable way to expand food production on the already overcrowded and environmentally degraded island. In fact, food production was starting to decline. There was no way to escape from the island, because there were no big trees left to build canoes from, and even if there were, there was no way for everyone to escape from the island. There was no way to get off, and nowhere to go if they could.

Easter Island is a very good analogy to the situation of the entire world today. I think we are getting close to the peak of global food production and population. I don't know that it will happen in this generation, but it will happen soon.

Back to Easter Island. For many generations before the collapse, the islanders were able to expand food production to keep up with population growth, Then, in a single generation, they suddenly went from abundance to scarcity. There simply wasn't enough food to go around, and as existing children grew up, they needed more food. The stark reality was that at least 5,000 people had to die.

What were their options? Well, they could have drawn lots to decide who would live and who would die, but that isn't the sort of thing that people are willing to do. (For a good reason: it is a terrible reproductive strategy.) They could have fought over resources in a scramble of each against all, but anarchy isn't stable. Why? Well, because if two people fo...
MATS Applications + Research Directions I'm Currently Excited About
by Neel Nanda — 2025-02-06T11:03:40.093Z

I've just opened summer MATS applications (where I'll supervise people to write mech interp papers) I'd love to get applications from any readers who are interested! Apply here, due Feb 28

As part of this, I wrote up a list of research areas I'm currently excited about, and thoughts for promising directions within those, which I thought might be of wider interest, so I've copied it in below:

Understanding thinking models

Eg o1, r1, Gemini Flash Thinking, etc - ie models that produce a really long chain of thought when reasoning through complex problems, and seem to be much more capable as a result. These seem like a big deal, and we understand so little about them! And now we have small thinking models like r1 distilled Qwen 1.5B, they seem quite tractable to study (though larger distilled versions of r1 will be better. I doubt you need full r1 though).

This is so understudied that I’m excited about even just rigorous black box investigations here, like seeing what happens when you intervene on the chain of thought

For example, when a model solves a hard maths problem by thinking a bunch until it finds the key idea, what’s going on? Did it just “get lucky” and every line was another roll of the dice for finding the key idea? Or was it steadily making progress towards that goal!

How deterministic are these models? If you take the first K lines of the CoT and regenerate it, do you get the same output?

My mental model is that when o1 is solving a maths problem it’s essentially exploring a “reasoning tree” - at each step in a solution, there’s a bunch of possible next steps. o1 chooses promising ones based on heuristics, and explores that branch of the tree, but with a bunch of reflexes to eg stop and say “that doesn’t sound right” or “I think I’ve gone wrong, let’s backtrack and try another path” or “let’s list out all relevant factors, and hope this helps me choose the next step better”

Can you find any evidence for or against this hypothesis?

An obvious safety strategy is “let’s just read the chain of thought and see if it thinks anything suspicious”. Can you find evidence for or against this being effective?

Eg, is the chain of thought faithful to the model’s underlying computation? Can you find evidence of unfaithfulness? What happens if you edit seemingly key steps in the chain of thought?

Can you create situations where the model does use suspiciou

...
Don't go bankrupt, don't go rogue
by Nathan Young — 2025-02-06T10:31:14.312Z

Would you bet your life’s savings on a 50% chance to triple it? Now imagine betting your sanity, values and goals. How much should you be willing to put on the table?

You wanna bet?

Let’s say you go to a casino, which offers a coin flip. Heads you triple it, tails you lose it. How much of your money should you bet on it? How many times should you do so?

One might think that this question is solely about expected value. Does it offer better return than other possible investments? If so, bet everything, if not, nothing.

This seems like a myopic view. For most of us, it’s about expected utility. How do I value very large amounts of money? What does one feel like if I go bankrupt? Because even if it’s the highest expected value option, if I put all my money on it, I’ll go bankrupt half the time.

Many people settle financially for only betting part of their wealth, even on outcomes they expect to be multiples of the money they put in. If you go bankrupt, you can’t play anymore, you’re stuck at zero. And for most people[1] the potential negatives outweigh the potential positives.

Your morality has a bank balance

I suggest there is a similar interplay between power and purpose. I can get more and more power, but I may be risking my integrity or goals and all that power isn’t much use if I am unrecognisable to myself.

Famously Sam Bankman-Fried said he would bet the whole world for a 51% chance of 2 worlds. Here is the tail end of that exchange[2]:

[Tyler] COWEN: Then you keep on playing the game. So, what’s the chance we’re left with anything? Don’t I just St. Petersburg paradox you into nonexistence.

BANKMAN-FRIED: Well, not necessarily. Maybe you St. Petersburg paradox into an enormously valuable existence. That’s the other option.

Bankman-Fried then propagated one of the largest frauds in history. He took a business worth billions, and in order to try and have it be worth tens of billions, ended up in jail.

I don’t understand Bankman-Fried, but I do wonder if his past self would endorse this. If we could take him back in time, stand him in front of his teenage self, whether that boy would endorse, or be appalled by, the man he had become[3].

Likewise, many look at Elon Musk and wonder where the autistic genius went, to be replaced by the right wing, edgelord industrialist[4].

“What good is it for someone to gain the whole world, yet forfeit their soul?" The Bible, ...
Voting Results for the 2023 Review
by Raemon — 2025-02-06T08:00:37.461Z

The votes are in for the 2023 Review!

6,264 posts were written in 2023

662 of them were nominated.

209 of them got at least one review, and a positive review-vote total.

50 of them shall be displayed in the Best of LessWrong, Year 2023.

Reviews

Exactly 100 people wrote reviews, and many of them I found particularly valuable. I want to give some particular shout outs to Ryan Greenblatt, John Wentworth, and Steve Byrnes.

I found Ryan Greenblatt's reviews to be sort of relentlessly reasonable. He reviewed many posts thoughtfully, clearly state flaws, value props, and concrete potential improvements. I particularly liked his review of Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust

I personally had the most direct "worldview update" from John Wentworth's The Case Against AI Control Research. I don't fully agree with John's take, but I found distinguishing how bad scheming is at different AI stages, and what else needs to be going right at that stage, a quite useful frame.

I was also grateful for Steve Byrnes extensive opinionated review of the “Sharp Left Turn” discourse.

Here's a general screenshot of top of the Review Leaderboard. Thanks to everyone who participated.

And, reminder, reviews now appear on the Spotlight Items, at the top of the home page, and on the /bestoflesswrong page.

Weighted Vote Totals

In past years, we calculated the votes based on two groups of users: Established users with 1000+ karma (who got 3x the vote weight), and users with <1000 karma.

This time, we're replacing that arbitrary cliff with a more granular "your review vote gets multiplied by your Strong Vote power." I mentioned we'd be doing something like that in the announcement post, noting:

I haven't been happy with the arbitrary cliff here. It gives more power than I really wanted to allocate to people with 1000 karma, and not enough weight to people who have been around much longer and have demonstrated good judgment. But, karma is still a pretty messy indicator, so I don't want to give too much power to high karma users either.

Ultimately it seemed that "Strong Vote power" basically did what we wanted. Very high karma users get more voting power, but each additional point takes roughly twice as much karma as the previous point, which limits how extreme the difference can get. Users pretty quickly ramp up to around ~6x voting power. Most longte...
Chicanery: No
by Screwtape — 2025-02-06T05:42:45.095Z

There is a concept I picked up from the Onyx Path forums, from a link that is now dead and a post I can no longer find. That concept is the Chicanery tag, and while I’ve primarily used it for tabletop RPGs (think Dungeons & Dragons) I find it applicable elsewhere. If you happen to be better at searching the forums than I am, a link would be lovely.

Chicanery:
Noun.
The use of trickery to achieve a political, financial, or legal purpose.
From French chicanerie, from chicaner ‘to quibble’.

I.

Imagine a magic power in a roleplaying game that says “You may change your features to look like another humanoid. You can change your entire appearance, even your sex, but cannot look like something significantly larger or smaller, or anything that isn’t humanoid.” Imagine it being positioned as a basic disguise power, not too high level.

Now imagine doing your level best to rules-lawyer this power and break it over your knee.

Can I change my features to have gold fingernails, clip my fingernails, and sell the gold? Can I change myself to look exactly like the queen? How much larger is “significantly” larger anyway, can I get tall enough for the extra reach to matter in a fight? Can I change into someone much more attractive and get a charisma boost? Can I change into a humanoid creature with claws, and cut myself free of these ropes I’m tied with? Can I change into a magic humanoid creature whose hair is on fire and have my hair be on fire, then light a match off it? Can I change into a humanoid creature with wings, then fly? Hey, I just got stabbed- can I change into me, but without the open wound? If so, does that heal me?

This kind of thing can be brainstormed by a good player in about five minutes, and a lot of it will be shut down by a skilled Dungeon Master just as fast. You can see the scar tissue of it in the fact that some versions of this power are

about a page long

, mostly disclaimers of what you can’t do. It seems excessive, especially since the basic disguise power is both a fun fantasy trope and also not that complicated. Wouldn’t it be great to have a rulebook that wasn’t three hundred pages long?

And yet, that kind of clever rules-lawyering can lead to some fun moments in roleplaying games.

Enter “Chicanery: No” and its partner, “Chicanery: Yes.”

II.

Here’s the idea:

When writing rules, game designers sometimes want you to ex...
BIDA Calendar iCal Feed
by jefftk — 2025-02-06T01:30:07.887Z

A while ago I played around with manually generating iCal feeds for
events, encoding the BIDA's regular schedule (1st and 3rd Sundays back
then, with a more complex pattern for family dances and open bands) as
in a machine-readable format. This was a fun project, but it didn't
work very well: it didn't have any way to handle cancellations,
special events, or other one-off adjustments to the schedule.

Still, having events automatically show up on calendars is great, and
after Jeff Keller asked
I decided to put something together.

Here's the feed: bidadance.org/events.ics.

You can add it in Google Calendar by going to "Other Calendars" >
"+" > "From URL":

And then pasting in the calendar URL:

Here's the next afternoon contra I'm able to attend,
showing up on my phone:

For the actual implementation a script
reads in the machine-readable list of scheduled events (dances.js).
This is a bit more awkward that it could be, since the list is
essentially JSONP
and not pure JSON, so it needs to do a little hacky parsing. Then it
interprets the records and uses the icalendar package to
build in iCal file. Dates, times, and libraries can all be a bit
tricky when they're an area you don't work in much, and I found Claude to be pretty helpful
here.

I also set up a cron job (script)
that looks for changes each hour and if so checks them in.

Let me know if you see any problems with it!
C'mon guys, Deliberate Practice is Real
by Raemon — 2025-02-05T22:33:59.069Z

I'm writing a more in-depth review of the State of Feedbackloop Rationality. (I've written a short review on the original post)

But I feel like a lot of people have some kind of skepticism that isn't really addressed by "well, after 6 months of work spread out over 1.5 years, I've made... some progress, which feels promising enough to me to keep going but is not overwhelming evidence if you're skeptical."

So, this post is sort of a rant about that.

A lot of people have default skepticism about "the feedbackloop rationality paradigm" because, well, every rationality paradigm so far has come with big dreams and underdelivered. I, too, have big dreams, and I am probably not going to achieve them as much as I want. That's not a crux[1] for me.

A lot of people also have some notion that "building multiple feedback loops" is a kind of elaborate meta that puts your head in the clouds. Dudes, the whole point of multiple feedback loops is to put you in direct contact with reality in as many ways as practical.

The thing that drives me nuts is, is that the basic premise here is:

If you put in a lot of effortful focused, practice...
...on the edge of your ability
...with careful attention to immediate feedback
...and periodic careful attention to "is this process helping my longterm goals?"

...you will get better at thinking.

(And, if a lot of people systematically do that and we track what works for them, we can get from the domain of Purposeful Practice to Deliberate Practice[2])

This seems so reasonable. This really is not promising a free lunch.

(I should clarify: I am not saying that there is robust scientific evidence for deliberate practice working for thinking. There is not – the evidence for meta-learning is generally confused and sucks and there particularly is not evidence that deliberate purposeful practice works on openended conceptual questions. I'm saying "c'mon, what the fuck do you think will work better at improving your ability to think? a) careful, focused practice on the edge of your ability with careful attention to feedback b) just generally trying to do object level work in domains where you ~never get much feedback on whether you're really accomplishing your ultimate goals?")

There are a lot of followup questions that aren't as obvious, but it seems crazy to not believe that. And if you took it seriously, I don't think the right ...