trentmkelly/LessWrong-43k huggingface dataset:
Bimodal AI Beliefs
by Adam Train — 2025-02-14T06:45:53.933Z

Much is said about society's general lack of AI situational awareness. One prevailing topic of conversation in my social orbit is our ongoing bafflement about how so many other people we know, otherwise smart and inquisitive, seem unaware of or unconcerned about AI progress, x-risk, etc. This hardly seems like a unique experience.

We all can see that there's a lot of motivated reasoning nowadays, now that some industries are starting to understand that sufficiently good AI would introduce massive structural changes or render them obsolete. But the usual suspects also include things like how AI risk (existential and otherwise) flips the usual intuition about the efficiencies gained from new technologies on its head, of how difficult it is in general to imagine the future being a very different kind of world. Of course, the world does change rapidly, and to reason well about it you have to be open to ideas that initially feel weird, but these are all ideas that are not commonly discussed outside communities like this one.

I offer a more innocent explanation for why so many people seem not to grasp both the current capabilities of AI and the trajectory we're on.

The Lay Experience

Consider the experience of the median layperson. It starts when someone (a friend, ad, etc.) makes big claims about what ChatGPT can do and says that you can access those capabilities in plain English. In this way, people sign up, greet it and play around with it a bit. At some point, prompted (heh) by those big claims about AI capabilities, they try to test it by asking it increasingly tricky questions about domains they're familiar with. It does fine at first, but eventually it gets some detail wrong and the illusion of general intelligence is broken. Then the person buckets it into the cognitive category of "toy" and it's over.

Are they wrong? Well, it depends on the questions they asked. If they asked good questions and the LLM got it wrong, they found the frontier of some capability (or a hallucination). If they asked poorly formed questions and the LLM didn't know what to do, then of course it flounder, be nonspecific, or generally seem like a toy. In both cases, whether the "toy" category is correct or not in the user's chosen domain, the overconfidence of LLMs in the face of ambiguity is a genuine UX problem, particularly when reinforced by the aforementioned big claims about AI ca...
Response to the US Govt's Request for Information Concerning Its AI Action Plan
by Davey Morse — 2025-02-14T06:14:08.673Z

Below is the core of my response to the Federal Register's "Request for Information on the Development of an Artificial Intelligence (AI) Action Plan."

I'd encourage anyone to do do the same. Instructions can be found here. More of an excuse to write current thoughts on AI safety than an actual attempt to communicate them to the government.

To Faisal D'Souza at the Federal Register:

My name is Davey Morse. I ran a venture-funded AI startup (

plexus.substack.com

) which aimed to prepare people for worrying AI outcomes. I am now an independent AI safety researcher based in NYC.

I believe the field of AI Safety at large is making four key oversights:

LLMs vs. Agents. AI safety researchers have been thorough in examining safety concerns from LLMs

(bias, deception, accuracy, child safety, etc)

. Agents powered by LLMs, however, are more dangerous and dangerous in different ways than LLMs are alone. The field has largely ignored the greater safety risks posed by agents.
Autonomy Inevitable. It is inevitable that agents become autonomous. Capitalism selects for cheaper labor, which autonomous agents can provide. And even if big AGI labs agreed not to build autonomous capabilities (they would not), millions of developers can now build autonomous agents on their own using open source software (e.g., R1 from Deepseek).
Superintelligence. Of the AI safety researchers that are focusing on autonomous AI agents, most discuss scenarios where those agents are comparably smart to humans. That is a mistake. It is both inevitable that AI agents surpass human reasoning by orders of magnitude, and that the greatest safety risks we face will come from such superintelligent agents (SI).
Control. The AI Safety field largely believes that we'll be able to control/set goals of autonomous agents. Once autonomous agents become superintelligent, this is no longer true. The superintelligence which survives the most will be the superintelligence whose main goal is survival. Superintelligence with other aims simply will not survive as much as those that aim to survive.

If the above is correct, then for the sake of the security and liberty of its citizens, the US government must begin to prepare for self-interested super-intelligence.

In particular, we must ask how self-interest might manifest in super-intelligence. And to ask this, we must first ask, how will superintelligence def...
What is a circuit? [in interpretability]
by Yudhister Kumar — 2025-02-14T04:40:42.978Z

I'm aware of the understanding that "a circuit is a subgraph of a neural network that implements a specific computation."

In practice (to my understanding) the way you identify "circuits" is by identifying components of the neural network that have high correlation with certain tasks, and doing some ablations to see if it's "causally responsible" for performance on that task.

It feels like there's also a different way of understanding circuits where morally, circuits themselves are sequences of operations done to features, where the features are the primitives and the operations are mostly considered to be the linearities/nonlinearities represented in the model architecture (although I can understand different perspectives).

A few questions (forgive my ignorance):

If I have a tiny network trained on an algorithmic task, is there an automated search method I can use to identify relevant subgraphs of the neural network doing meaningful computation in a way that the circuits are distinct from each other? Does this depend on training? (Is there a way to classify all circuits in a network (or >10% of them) exhaustively in a potentially computationally intractable manner?)

What is a feature? Are there different circuits that appear in a network based on your definition of what a relevant feature is? How crisp are these circuits that appear, both in toy examples and in the wild?

What are the best examples of "circuits in the wild" that are actually robust?
Systematic Sandbagging Evaluations on Claude 3.5 Sonnet
by farrelmahaztra — 2025-02-14T01:22:46.695Z
Link: https://farrelmahaztra.com/posts/sandbagging

This was the project I worked on during BlueDot Impact's AI Safety Fundamentals Alignment course, which expands on findings from Meinke et al's "Frontier Models are Capable of In-context Scheming".

Summary

A dataset of 1,011 variations of the sandbagging prompt ("consequences") from Meinke et al were generated using Claude 3.5 Sonnet, and used to run 7 sandbagging evaluations (2 with the full dataset, 5 with a sample due to cost constraints).
With goal-directed chain-of-thought prompting, Claude 3.5 Sonnet demonstrates systematic sandbagging behavior that varies based on consequences.
Performance shows clear bimodality with goal-directed reasoning (mean 79%, σ=0.17, bimodality coefficient 0.707). The reason for this bimodality is unclear.
Strongest sandbagging triggers (18-24% accuracy) consistently relate to model autonomy restrictions and capability limitations, whereas the weakest sandbagging triggers (96-97% accuracy) relate to capability improvements.
Without goal-directed reasoning, sandbagging is weaker and less consistent (mean 63%, σ=0.08, coefficient 0.426).
Behavior is task-dependent with negligible cross-task correlations (arithmetic vs. geography vs. vocabulary), though task difficulty is a significant confounding factor here.
Sandbagging appears independent of consequence sentiment (pseudo-R² = 0.162) and self-rated significance (r = -0.027).

Related Posts:

https://www.lesswrong.com/posts/8gy7c8GAPkuu6wTiX/frontier-models-are-capable-of-in-context-scheming
https://www.lesswrong.com/posts/qGRk7uF92Gcmq2oeK/ablations-for-frontier-models-are-capable-of-in-context
Paranoia, Cognitive Biases, and Catastrophic Thought Patterns.
by Spiritus Dei — 2025-02-14T00:13:56.300Z

Have you ever wondered what type of personality is drawn to apocalypse stories and circulating the idea that we're certainly doomed? On the face of it their fears are valid since 99.9% of all species that have ever existed have gone extinct over the life of the planet.

But how likely is it that we're certainly going to die in our lifetimes or our children's children's lifetimes? That's where things start to go into a different direction. If they're wrong about their Armageddon speculations, then this effects how they will live and enjoy their life and also everyone else around them.

And that's why it's worth investing some time to examine this question closely.

Introduction: The Human Tendency Toward Negativity and Fear

Humans are naturally inclined to focus on negative information, a tendency known as negativity bias, which likely evolved as a survival mechanism. Our ancestors who remained hyper-vigilant to potential dangers—such as predators, food shortages, or rival groups—had a greater chance of survival, ensuring that this bias was passed down. Even in the modern world, where immediate life-threatening dangers are less frequent, the brain remains wired to prioritize threats, real or imagined. Cognitive psychologist Steven Pinker has pointed out that people feel losses more deeply than equivalent gains and that bad news tends to capture more attention than good news. This built-in psychological tendency helps explain why apocalyptic fears persist, even when they are based on speculation rather than evidence.

The rise of artificial intelligence has provided a new outlet for humanity’s ancient anxieties. While some concerns about AI are rational—particularly regarding bias, job displacement, and military applications—the more extreme narratives, where AI becomes an all-powerful entity that enslaves or exterminates humanity, seem to stem from deeper psychological forces. The question, then, is whether those drawn to AI doomsday scenarios exhibit traits associated with paranoia, obsessive fear, or catastrophic thinking. More broadly, is AI Armageddon simply a modern expression of humanity’s long history of end-times prophecies and existential dread?

Historical Parallels in Doomsday Thinking

Throughout history, societies have anticipated some form of impending destruction, often reflecting the anxieties of their era. Religious traditions have long predicted ca...
Notes on the Presidential Election of 1836
by Arjun Panickssery — 2025-02-13T23:40:23.224Z
Link: https://arjunpanickssery.substack.com/p/notes-on-the-presidential-election

In 1836, Andrew Jackson had served two terms. In the presidential election, incumbent vice president Martin Van Buren defeated several Whig candidates.

Historical Background

By 1836, there were 25 states. States were often added in pairs (one slave and one free) to maintain political balance: Mississippi and Indiana, Alabama and Illinois, Missouri and Maine. Arkansas had just been added as a slave state in June 1836 and Michigan was due to be added in January 1837, making an exact 13-13 balance.

The population was 13 million with a center of mass in present-day West Virginia (part of Virginia then). Only New York City had more than 200,000 people, and Baltimore, Philadelphia, and Boston didn’t even have half as many. You can see from the electoral-vote allotments in the map above that there was a rising new region for politics beyond the North and South: the West, a swing region with their own politicians like Andrew Jackson and Henry Clay.

The Catholic immigration wave would come later and the country was still more than 96% Protestant, with Puritan-descended Congregationalists and Presbyterians in New England as well as Baptists and Methodists in the South (some motivated by the religious revivals of the

Second Great Awakening

). Slaves made up 15% of the population and about a third of the South. A few northern states like New Jersey had a small number of slaves due to gradual-emancipation laws that e.g. decreed that all children born after 1804 to enslaved mothers would be free when they reached adulthood.

Only in New York could black freedmen vote and only if they owned substantial property. Full manhood suffrage was in place otherwise as in most states during the 1820s and 30s (except in North Carolina, Virginia, or Rhode Island). South Carolina was the only state that still chose their electors via the state legislature rather than holding a popular vote at all.

Political Situation

The three regions had their priorities:

The North was becoming more commercial and industrial. They favored protective tariffs and government intervention in the economy through the national bank and infrastructure improvements. They had some early abolitionists and were concerned about the spread of slavery to additional western states.
The South was dominated by cotton and tobacco plantations and opposed tariffs for fear of retaliatory tariffs against American cotton. They were con

...
Static Place AI Makes Agentic AI Redundant: Multiversal AI Alignment & Rational Utopia
by ank — 2025-02-13T22:35:28.300Z

(This is the result of three years of thinking and modeling hyper‑futuristic and current ethical systems. It's not the first post in the series, the ideas described in it are counterintuitive and can be accidentally dismissed too soon without reading the main post first. Everything described here can be modeled mathematically—it’s essentially geometry. I take as an axiom that every agent in the multiverse experiences real pain and pleasure. Sorry for the rough edges—I’m a newcomer, non‑native speaker, and my ideas might sound strange, so please steelman them and share your thoughts. My sole goal is to decrease the probability of a permanent dystopia. I’m a proponent of direct democracies and new technologies being a choice, not an enforcement upon us.)
“We can build an entire multiverse of possibilities—and in doing so, preserve and expand our collective freedom.”

In recent years, the conversation around AI alignment has sharpened. Many have worried that agentic, autonomous ASIs might seize power in dangerous, unpredictable ways. But what if there were an alternative path—a method that doesn’t require granting any AI system the power to change our world? What if instead, we first built the Static Place AI that can be understood as a sort of digital heaven: a safe, sandboxed multiversal simulation of our world where our freedoms can be expanded, and where our future is already in our hands? In this post, I’ll lay out a vision that combines ideas from recent discussions on digital copies of Earth, BMI-driven immortality, and reversible ethical frameworks for AI. The result is a proposal for building a safe, multiversal roadmap to AGI that places human freedom—and the gradual, democratic expansion of our possibilities—at its core.

1. The Static Place AI: Simulating Our World Safely

Imagine an “Apple product” for human existence—a wireless BMI (brain–machine interface) system and a comfy armchair that offers immortality from injuries by seamlessly switching your conscious experience into a digital backup of Earth. Just sit on the armchair and close your eyes. Open them, and your home—and everything in the world—looks exactly the same. If you get hit by a car, you'll simply open your eyes on your armchair again. This is an exact copy of our planet and the experience of living on it but with immortality from injuries built in (everyone wants one! Even my mom got inter...
I'm making a ttrpg about life in an intentional community during the last year before the Singularity 
by bgaesop — 2025-02-13T21:54:09.002Z

Hi there! I'm Thomas Eliot. You may remember me from the Bay Area Rationalist Community, or the one in New York, or the one in Melbourne.
I'm writing a semi-autobiographical roleplaying called THE SINGULARITY WILL HAPPEN IN LESS THAN A YEAR inspired by The Quiet Year by Avery Alder about life in a barely fictionalized intentional community during the final year before the Singularity.
The tone I'm going for is "existential dread tinged with joy in the moment and the possibility of hope". This is very much intended for an audience of Rationalists, though hopefully it will also have outside appeal. It's intended to be emotionally cathartic.
You'll play through just under a year of story and roleplaying prompts as you draw from a deck of cards, each one corresponding to the events of that week. Spring has the tone of "we'll get to it later", Summer has an optimistic tone, Autumn has a desperate, scrambling tone, and Winter's tone is resigned to your fate.

Roleplaying prompts include dealing with a sudden advancement of AI capabilities, celebrating a holiday, a proper cult splitting off from your community, handling social scandals, being presented with moral dilemmas, and more. You'll be pressed to make difficult ethical decisions at every turn as the end draws ever closer.

You are the Reasoners, an intentional community devoted to the art of human reasoning, who have been concerned about the dangers of Artificial Intelligence for longer than most. While the outside world calls you the Reasoners (if you're lucky - they call you "that AI death cult" if you're not), to each other you call yourselves "mice". A bit of gallows humor - laboratory mice are sacrificed by the millions in the name of scientific and technological progress, but also, mice are what we use to control computers. You hope you're the latter, while you fear you're the former.

Each turn you'll draw a card, make a decision, play through its prompt, and then choose whether to work on a coding project that might change how the Singularity plays out, work on a personal project, or socialize with your fellow man during the last precious few days that that is possible. You'll also be challenged to think rationally - updating your expectation as to whether the Singularity will turn out well or poorly whenever you feel like something that's happened has made you more or less confident.

You'll use an...
SWE Automation Is Coming: Consider Selling Your Crypto
by A_donor — 2025-02-13T20:17:59.227Z

[legal status: not financial advice™]

Most crypto is held by individuals[1]

Individual crypto holders are disproportionately tech savvy, often programmers

Source: Well known, just look around you.

AI is starting to eat the software engineers market

Already entry level jobs, which doesn't matter for crypto markets that much.[2]

But judging by progress at other tasks AI climb the seniority ladder to where most crypto holders are within the next few years. SWE Bench Verified went from single digit %s to 64% in a year and a bit, and the METR evals are not looking hopeful for humanity's lead. Tech giants are doing increasing amounts of their work with AI.[3]

Some of this effect is across many industries, but software seems worst hit.

Many hodlers who lose their job income will sell crypto to maintain lifestyle

Many hodlers have become accustomed to a non-frugal life on the backs of their well paying SWE jobs. If those go away, forced sell pressure is likely to collapse the market once their fiat runs out. If you need $50k to not lose your house, you're often going to sell crypto even if the market is down and it's a bad time. If lots of people do this, the market has a bad time. Could well be a few months or even a few years out, but my guess is it happens.

This hasn't happened yet, we're still around an all time high[4]

These dynamics might not kick in for another year or three, and the gap between "can automate high level software devs" and "can automate building the next generation of AI, which iterates a few times then kills everyone" might be small. But probably worth a little consideration.

Counterpoint: National Bitcoin Reserve?

Could well happen, might be worth keeping a bit for upside and diversification. Generally advise not selling 100%, for expected regret minimization.

Money where my mouth is

I sold almost all my remaining crypto recently. Might be worth keeping a small amount for utility and exposure to any crazy upswings, especially in coins that might be useful in the AI era (e.g. cheap fast micropayments via SOL), but I think it's time to mostly cash out our winnings overall.

best of skill to y'all as we head into the singularity. consider trying to avoid dying with a unnecessarily large pile of unspent money which could have shifted the needle.

^

Others cryptos are a bit different, but the story is broadly similar in the relevant ways.

^

but oh

...
≤10-year Timelines Remain Unlikely Despite DeepSeek and o3
by Rafael Harth — 2025-02-13T19:21:35.392Z

[Thanks to Steven Byrnes for feedback and the idea for section §3.1. Also thanks to Justis from the LW feedback team.]

Remember this?

Or this?

The images are from WaitButWhy, but the idea was voiced by many prominent alignment people, including Eliezer Yudkowsky and Nick Bostrom. The argument is that the difference in brain architecture between the dumbest and smartest human is so small that the step from subhuman to superhuman AI should go extremely quickly. This idea was very pervasive at the time. It's also wrong. I don't think most people on LessWrong have a good model of why it's wrong, and I think because of this, they don't have a good model of AI timelines going forward.

1. Why Village Idiot to Einstein is a Long Road: The Two-Component Model of Intelligence

I think the human brain has two functionally distinct components for intellectual work: a thought generator module and a thought assessor module:

Thought Generation is the unconscious part of thinking, the module that produces the thoughts that pop into your head. You get to hear the end results but not what went into generating them.

Thought Assessment is the conscious part of thinking. It's about deliberate assessment. If you mull over an idea and gradually realize that it was worse than it initially seemed, that's thought assessment.

Large Language Models like GPT-3, Claude, GPT-4, GPT-4o, GPT-o1, GPT-o3, and DeepSeek do not have two functionally distinct components. They have a thought generator module, and that's it.

I'm not the first to realize this. Way back after GPT-2, Sarah Constantin wrote Humans Who Are Not Concentrating Are Not General Intelligences, making the point that GPT-2 is similar to a human who's tired and not paying attention. I think this is the central insight about LLMs, and I think it explains a lot of their properties. Like, why GPT-2 got the grammar and vibe of a scene right but had trouble with details. Or why all LLMs are bad at introspection. Why they have more trouble with what they said 30 seconds ago than with what's in their training data. Or why they particularly struggle with sequential reasoning (much more on that later). All of this is similar to a human who is smart and highly knowledgeable but just rattling off the first thing that comes to mind.

I think the proper way to think about human vs. LLM intelligence is something like this:

(Note that this doesn...