trentmkelly/LessWrong-43k huggingface dataset:
Become a Superintelligence Yourself
by Yaroslav Granowski — 2025-05-10T20:20:30.157Z

"Crystallect" stands for the crystal transparency and order in empowering the user’s intellect in contrast to the black box of neural-network-based Artificial Intelligence.

Nerds and Bullies

Kids in school, as I remember it, had two ways for getting good grades: to study hard or to bully some nerd kid into doing their homework. The second seemed more rational in the short-term. And sometimes in the long-term too, because some of those bullies learned soft skills and became managers of those hard-studying nerds.

Specialization and division of labor made it common sense to think this way: If something seems hard for you, let someone else do the job. And even the most nerdy among us like the idea of Artificial Superintelligence solving all problems for them if they believe they can manipulate it into compliance.

I believe, they can't. LessWrong contains plenty of materials on the problem of AI Alignment, so I'll not elaborate on that. Instead, I wanted to talk about the possibility of the first way: to study hard and become a Superintelligence yourself, however unrealistic it may sound. Let us fantasize first and discuss technicalities later to see where we, as a humanity, may be heading.

Power

Imagine an ultimate picture of you with your brain interfaced with some high-tech increasing your intellectual capabilities and data throughput. Such intellectual capabilities that you could contemplate quantum gravity theories or analize the genome of your cat. Such data throughput that you could manage in real time swarms of combat drones in the battlefield like it were your own fingers. Or you could manage all cyber-servants of your international restaurant chain as if you were the every one of them.

I've mentioned military application for a reason. New technologies give an edge in the power struggle, and thus attract power players. Intelligent people usually avoid direct participation in conflicts. They invent weapons but let harsh military men do the dirty job. This will be not the case anymore.

Such technology eliminates the need for specialization and labor division. If you have not enough resources to solve some problem and need the help of your neighbour, you may as well devour his resources and do the job yourself. This sounds pretty dangerous, isn't it? Like in the "Highlander" movie, leading to the survival of a singular, most aggressive superintelligence.

The hope fo...
A Look Inside a Frequentist
by Eggs — 2025-05-10T15:18:02.796Z

Today I experienced the Sequences post Beautiful Probability for the first time. I will begin by quoting the quotation that Eliezer began with:

Let me introduce this issue by borrowing a complaint of the late great Bayesian Master, E. T. Jaynes (1990):

"Two medical researchers use the same treatment independently, in different hospitals. Neither would stoop to falsifying the data, but one had decided beforehand that because of finite resources he would stop after treating N=100 patients, however many cures were observed by then. The other had staked his reputation on the efficacy of the treatment, and decided he would not stop until he had data indicating a rate of cures definitely greater than 60%, however many patients that might require. But in fact, both stopped with exactly the same data: n = 100 [patients], r = 70 [cures]. Should we then draw different conclusions from their experiments?" (Presumably the two control groups also had equal results.)

This quote captured me completely—I found myself caught up thinking through the implications before I could even finish reading the rest of the post. It gave me an unexpected insight into the internal logic and motivations behind the frequentist approach to statistics. There is a genuine difference in the information conveyed by the two scenarios Jaynes describes.

To make the discussion more approachable, let's introduce names. Let's call the first researcher George. He stopped after treating 100 patients regardless of the results. The second researcher, whom we'll call Bessel, continued until the results supported his belief. This small act of naming helps us think about their mindsets and intentions. Similarly I'm going to call Frequentist the method that assigns to the experiments different p-values, and Bayesian the method that assigns them the same likelihood ratios.

The Frequentist Approach

Computing p-values is what Mr. Frequentist is all about. This method is centered on pre-defined procedures that allow us to interpret data in a principled and repeatable way: "What is the chance that using the same method would give this result under the null hypothesis?" Necessarily this approach says a lot about the method being used. While this mindset may seem flawed, it reflects an important goal: measure the a priori expectations of how often a method will yield false positives.

When the rules are followed, the f...
Open-source weaponry
by samuelshadrach — 2025-05-10T13:11:10.623Z
Link: https://samuelshadrach.com/raw/text_english_html/my_research/open_source_weaponry.html

2025-05-10

If we did end up with a world where even its most powerful organisations cannot keep secrets, then values and capabilities of all orgs become public.

Values becoming public means the public can decide if the org represents their true values, and if not, coordinate to shut down the org or build another org to replace it.

This is control via a more direct democracy than what we currently have, and not control via market.

Capabilities becoming public means all weapons capabilities will also be open source. This document analyses implications of open-source weaponry.

Examples of technologies that will become open source.

nuclear weapons and ICBMs, underlying supply chains for uranium/plutonium enrichment, missile guidance systems etc

bioweapons and underlying supply chains for biotech

AI and GPUs and underlying supply chains

(hypothetically) ASI, BCIs, human genetic engg, human connectome simulations

(hypothetically) gene drives, solar geoengineering, etc

(hypothetically) any offence-beats-defence weaponry, including such weaponry deployable by small groups

As of 2025, I am okay with all these technologies being open sourced as long as this is a byproduct of the organisation's other information (such as values and decision-making processes) also leaking.

Why is open sourcing weaponry not fatally unsafe?

Open source nuclear weapons

As of 2025-05 only 9 governments have nuclear weapons. Many other governments would like to have them but don't. My guess is that the bigger causal factor by far for this is that they're pre-emptively threatened by (some subset of) these 9 governments, not that they lack the technical knowledge and infrastructure to build nuclear weapons.

As of 2025-05, my guess is open sourcing technical knowledge to produce nuclear weapons is not likely to significantly increase the number of governments in the world that have nuclear weapons.

Open source bioweapons

As of 2025-05, most bioweapons require an industrial scale production process to manufacture. This keeps them out of the reach of smaller organisations, and governments are by default aware of and responsible for the biotech supply chains in their countries.

DNA synthesis machines require an industrial process to manufacture as of 2025-05. Genetically engineered bioweapons can likely be more deadly than bioweapons without it, and these are reliant on DNA sy

...
Glass box learners want to be black box
by Cole Wyeth — 2025-05-10T11:05:23.003Z

Epistemic status: intuitive speculation with scattered mathematical justification.

My goal here is to interrogate the dream of writing a beautiful algorithm for intelligence and thereby ensuring safety. For example:

I don't know precisely what alternative he had in mind, but I only seem to remember reading clean functional programs from MIRI, so that is one possibility. Whether or not anyone else endorses it, that is the prototypical picture I have in mind as the holy grail of glass box AGI implementation. In my mind, it has careful comments explaining the precisely chosen, recursive prior and decision rules that make it go "foom."

Is the "inscrutable" complexity of deep neural networks unavoidable? There is has been some prior discussion of the desire to avoid it as map-territory confusion, and I am not sure if that is true (though I have some fairly subtle suspicions). However, I want to approach this question from a different angle.

And most importantly, I want to ask what this has to do with AI safety. Would an elegant implementation of AGI be safe? What about an elegant and deeply understood implementation?[1]

In a previous essay, I considered the distinction between understanding or discovering core algorithms of intelligence and searching for optimality guarantees. It occurs to me that the former is really a positive, creative exercise while the later is more negative. Or, perhaps it is more accurate to say that an optimal method must sit at the boundary of the positive and the negative; it is exactly as good as possible, and anything better would be impossible.[2]

Sometimes, there is no meaningfully optimal element in a class. For instance, there is no computable predictor which always learns to do as well as every other computable predictor on all sequences.[3] This is a failure of a relative notion of optimality. But perhaps it is more important to ask for good performance on some sequences than others. I think rationalists tend to (hyper?)focus on consistency-based conditions like "Bayes optimality," which I am not really sure can be viewed as optimality conditions at all (though they are certainly desirable to satisfy, all else held equal). Perhaps it is useful to think of the relative notions as external-facing and the consistency notions as internal-facing. I don't want to be too specific here about what makes for a good ...
Takes and loose predictions on AI progress and some key problems
by zef — 2025-05-10T10:11:52.757Z
Link: https://halcyoncyborg.substack.com/p/takes-and-loose-predictions-on-ai

Thanks to Gavin, Kaivu, Kushal, Nikola for giving feedback.

AI is moving fast. I wanted to put down some thoughts on the future and what I believe will be important, moreso on a technical level than a governance/policy perspective. Give me feedback, push back if you have takes!

What remains as a blocker to AGI?

ability to do long horizon tasks and reason on lots of information in context

I predict the memory problem will be solved in up to 2 years with RL training on long horizon environments. The bottleneck is more on data than ideas, and it's a lot of work but not a lot of insight.

creativity and taste

it seems that o3 is a step up, probably due to much better ability to dynamically query and reason on information

this is hard to formalize but important for research, along with general skills like understanding what matters and prioritizing, being able to debug the world and be reliable

it's plausible that the type of cognition for coming up with new ideas, doing scientific innovation, noticing discrepancies in a research process, understanding the current level frontier of knowledge, is not being represented / trained on and is hard to bootstrap

This remains nebulous, but it’s plausible that models could learn to bootstrap environments and data to develop these skills.

could take a while though

reasoning in non verifiable environments, "true" generalization

to what extent is true meta learning happening? I think to a large extent, real, long horizon RL with memory and result aggregation will be a large step forward on this because the model will actually have time to properly reason on the environment

but it will require environments that require the right kinds of skills - do we have those? can we generate them? maybe

then fundamentally with our current methods it's a data issue, and probably an expensive one

could also be solved by new algorithms, but I am less optimistic about this in the short term

very unclear

In terms of judging works like AI 2027, or other strong crazy AI soon believers, I think there are these key open questions:

to what extent does research automation accelerate the future of machine learning?

no strong gut feeling here, there is a lot of low hanging fruit in the research process to automate

but if you believe that we are not making progress on a fundamental problem, eg improving at modelling and predicting real s

...
Corbent – A Master Plan for Next‑Generation Direct Air Capture
by Rudaiba — 2025-05-10T04:09:58.593Z

When was the last time you thought about the CO₂ in the air you're breathing right now? For me, it was this morning. Why? Because we've hit 420 parts per million(ppm), higher than at any time in the past 3 million years[1].

Even if we stopped all emissions today, that legacy CO₂ would continue warming our planet for centuries. The climate math is clear: we need to actively pull carbon back out of the sky.

Executive Summary

Corbent builds shipping-container-sized "carbon vacuums" that capture CO₂ from air using 90% less energy than current methods. Our breakthrough combines metal-organic frameworks with electrochemical triggering to slash costs from $500+ to under $100 per ton. We're seeking strategic partners and funding to scale from our 2025 pilot to gigaton removal capacity by 2040, delivering carbon credits from the most energy-efficient direct air capture technology on the market.

Quick Specs:

Energy use: 0.65 GJ/ton CO₂ (vs. current DAC at 5–10 GJ/ton)
Module capacity: 500 tons CO₂/year per 40-ft container
Target cost: <$50/ton at scale (vs. $125–335/ton for current tech)
Storage method: Permanent basalt mineralization (>95% in 2 years)
Scaling approach: Factory-produced modules, roll-to-roll MOF sheets
Technology Readiness: Currently TRL 5, piloting TRL 6 in 2025

Why Today's Carbon Capture Falls Short

Here's why the DAC technology we have today simply won't scale to what we need:

It's an energy hog. CO₂ in air is just 0.041% – over 100 times more dilute than in industrial flue gases[2]. Current DAC systems demand a staggering 5–15 GJ of energy per tonne (1.4–4.2 MWh)[3].

This is equivalent to what an average U.S. home uses in 6–18 months. With this energy intensity, large-scale deployment becomes nearly impossible.

Current direct air capture methods demand on the order of 5–10 GJ per tonne CO₂ (heat + electricity combined), far above theoretical minimums. Alternative approaches and new materials could cut this dramatically – by up to an order of magnitude[4]. This figure illustrates the energy required per ton for today's solid DAC vs liquid DAC vs Corbent's electro-swing MOF approach, highlighting ~90% energy reduction[5].

It costs too much. Leading DAC companies charge $500–$600 per ton[6], with small-scale operations selling CO₂ removal for $1,000+ per ton[7]. The industry targets $100/t as the holy grail[8],...
What if we just…didn’t build AGI? An Argument Against Inevitability
by Nate Sharpe — 2025-05-10T03:37:22.084Z
Link: https://natezsharpe.substack.com/p/what-if-we-justdidnt-build-agi

Note: During my final review of this post I came across a whole slew of posts on LessWrong and the Effective Altruism Forum from several years ago saying many of the same things. While much of this may be rehashing existing arguments, I think the fact that it’s still not part of many mainstream discussions means it’s worth bringing up again. Full list of similar articles at the end of the post, and I'm always interested in major things I'm getting wrong or key sources I'm missing.

After a 12-week course on AI Safety, I can't shake a nagging thought: there's an obvious (though not easy) solution to the existential risks of AGI.

It's treated as axiomatic in certain circles that Artificial General Intelligence is coming. Discussions focus on when, how disruptive it'll be, and whether we can align it. The default stance is either "We'll probably build it despite the existential risk" or "We should definitely build it, because [insert utopian vision here]".

But a concerned minority (Control AI, Yudkowsky, Pause AI, FLI, etc.) is asking: "What if we just... didn't?" Or, in more detail: "Given the unprecedented stakes, what if actively not building AGI until we're far more confident in its safety is the only rational path forward?" They argue that while safe AGI might be theoretically possible, our current trajectory and understanding make the odds of getting it right the first time terrifyingly low. And since the downside isn't just "my job got automated" but potentially "humanity is no longer in charge, or even exists", perhaps the wisest move is to collectively step away from the button (at least for now). Technology isn't destiny; it's the product of human choices. We could, and I’ll argue below that we should, choose differently. The current risk-benefit calculus simply doesn't justify the gamble we're taking with humanity's future, and we should collectively choose to wait, focus on other things, and build consensus around a better path forward into the future.

Before proceeding, let's define AGI: AI matching the smartest humans across essentially all domains, possessing agency over extended periods (>1 month), running much faster than humans (5x+), easily copyable, and cheaper than human labor. This isn't just better software; it's a potential new apex intelligence on Earth. (Note: I know this doesn’t exist yet, and its possibility and timeline remain op...
Mind the Coherence Gap: Lessons from Steering Llama with Goodfire
by eitan sprejer — 2025-05-09T21:29:35.232Z

TL;DR

Context. Feature‑steering promises a cleaner, more interpretable way to shape LLM behavior than other methods, such as plain prompting. But does it actually work better?

What I did. Benchmarked Goodfire's Autosteer method against three others (Simple Prompting, Agentic, Combined Approach) + a control, on 10 steering goals × 30 prompts for two Llama variants (8B & 70B), and scored every output on Elicited Behavior Strength and Coherence Drop using an LLM‑as‑judge rubric.

Key findings.

Prompting ≈ best overall. Plain textual instructions already hit strong behavior scores without harming coherence.
Auto Steer ⇩ coherence. Stand‑alone steering drops coherence by ≈ 0.6 points and still under‑delivers on the target behavior.
Combined > stand‑alone. Simple Prompting +  Auto Steer gives the largest behavior boost (+0.4 points) but still inherits the coherence hit.
Manual feature selection beats Autosteer. LLM‑assisted (Agentic) feature picking outperforms Auto Steer on both metrics, for the largest model.

Take‑away. For now, prompting remains the cheapest, most reliable control knob; feature‑steering looks promising but needs smarter feature selection and coherence‑preserving edits.

Future work. In a follow-up post, I plan on improving the evaluation methodology, developing a more robust and aligned benchmark for measuring the value of using feature steering for safety-related scenarios.

All code & raw scores →

GitHub repo

.

Disclaimers.

This post is meant as a proof‑of‑concept and a snapshot of my research workflow for a week-long project, not a definitive judgment on steering methods. Treat the numbers as directional signals, and feel free to tear the setup apart—feedback (or PRs) welcome!
Goodfire's SDK has been updated since the time I ran the experiments (Jan. 2025), so results may not be up-to-date.

Motivation

Large Language Models (LLMs) are increasingly being deployed in real-world applications, from customer service to high-stakes medical contexts. However, ensuring these models behave consistently and reliably remains a fundamental challenge in AI alignment. This is particularly critical in safety-sensitive domains, where model hallucinations or inconsistent behavior could have serious consequences. While prompt engineering and fine-tuning is the current standard for controlling model behavior, its trial-and-e...
My Experience With EMDR
by Sable — 2025-05-09T21:25:39.111Z
Link: https://affablyevil.substack.com/p/my-experience-with-emdr

EMDR stands for Eye Movement Desensitization and Reprocessing, itself a subset of Bilateral Stimulation Therapy. Bilateral Stimulation Therapy is not well understood, but the gist is that activating both brain hemispheres (hence bilateral) helps people process traumatic memories. It’s increasingly being used in treating PTSD, but you don’t have to have severe trauma for it to be helpful.

I’ve been doing EMDR for around two years now in therapy, and it’s worth talking about.

Disclaimer: I am not a psychologist or therapist. I am a patient talking about my own experiences. None of this should be taken as me telling anyone to do anything.

Content Warning: I don’t usually do content warnings, but this is some sensitive stuff. Detailed discussion of psychological therapy and the mental wounds it is meant to treat. Some references to anorexia and PTSD as non-personal examples.

Background

We all have memories that evoke strong emotions in us. Some of those memories are sources of happiness: a wedding, the birth of a child, a major success or a cherished evening spent with the people you love. Sometimes the memory is from a source of pain: a breakup, a failure, a loss. These memories affect us - present tense. They’re tied in to how we perceive ourselves and the world.

Wounds That Never Healed

We have concepts in language and story for physical wounds that never fully healed. There’s the broken leg that leaves someone with a limp for the rest of their life, or the old scar that aches when it rains. We might talk about sucking venom from a snake bite or needing to clean out a cut before bandaging it so infection doesn’t set it. We’re aware that wounds need air, that what doesn’t kill you might make you stronger but it can also leave you crippled.

We have concepts in language and story for past misdeeds that come back to haunt people. We’ll say a politician has skeletons in their closet, or that nothing stays buried forever.

But we don’t have as many concepts in language and story for the idea of mental and emotional wounds that never quite healed. Sure, characters in a story have character flaws, and those flaws might be born from important moments in their life, the way that Anakin Skywalker was so traumatized by the loss of his mother that he was willing to commit unspeakable acts to prevent Padme’s death. But by and large character flaws are seen as baked-in, something int...