trentmkelly/LessWrong-43k huggingface dataset:
Expectation = intention = setpoint 
by jimmy — 2025-06-09T17:33:37.809Z

When I was first learning about hypnosis, one of the things that was very confusing to me is how "expectations" relate to "intent". Some hypnotists would say "All suggestion is about expectation; if they expect to have an experience they will", and frame their inductions in terms of expectation (e.g. "Your eyelids will become heavy"). The problem with this is that "I don't think it's gonna work". Other hypnotists would avoid this issue entirely by saying "I don't care if you think it will work. Follow my instructions, and you will get the results regardless of what you believe" and then say things like "Make your eyelids heavy". The problem with this is that "I don't know to do that!", which would be avoided by saying "You don't have to 'do' anything; I'm telling you what is going to happen, and your job is simply to notice when it does" -- back to square one. This dual path thing always confused me, because "Sure, we can equivocate and play sleight of mouth in order to trick people into getting the results[1], but which is it really?".

It's both.

They're the same thing.

In the same way that "running away from a tiger" and "running to catch a bus" are both the same object level behavior being used for a somewhat different purpose.

Initially, it seemed weird to me that humans are designed so that expectation and intent are implemented in the same way. Strange design choice, given that it creates massive vulnerabilities, where if we can be tricked into expecting we won't be able to remember something, it becomes self fulfilling.[2]

Turns out, it has to be this way - not just for humans, but for any system that tries to control anything. To see why, let's start with something simpler than a human mind.

Consider a thermostat.

When anthropomorphizing a thermostat, you might be tempted to model this controller as having "beliefs" and "desires". The output of the thermometer is the "belief" the system has about the temperature of the room, and the set point we dial to is the "desire" of the system, to which it attempts to regulate external reality. Under this model, it kinda seems like intentions are different kinds of things from expectations, because expectations are like beliefs which are about what reality is while intentions are like desires which are about what we want it to be, but isn't. But a thermometer reading isn't really an expectation...
Identifying "Deception Vectors" In Models
by Stephen Martin — 2025-06-09T17:30:22.634Z
Link: https://arxiv.org/pdf/2506.04909

Using representation engineering, we systematically induce, detect, and control such deception in CoT-enabled LLMs, extracting ”deception vectors” via Linear Artificial Tomography (LAT) for 89% detection accuracy. Through activation steering, we achieve a 40% success rate in eliciting context-appropriate deception without explicit prompts, unveiling the specific honesty related issue of reasoning models and providing tools for trustworthy AI alignment.

This seems like a positive breakthrough for mech interp research generally, the team used RepE to identify features, and were able to "reliably suppress or induce strategic deception".
Policy Design: Ideas into Proposals
by belos — 2025-06-09T17:26:45.414Z
Link: https://bestofagreatlot.substack.com/p/policy-design-ideas-into-proposals

This post on Best Of A Great Lot is a part of a series on the subject of designing a new form of governance. Each piece aims to stand alone, but fits together on the

Table of Contents

.

Previous:

A Sketch of Belocracy,

Evaluation as Feedback Cycle

,

Idea Generation and Sifting

,

The Belocrat

. Next: Executive Belocracy

So far in our implementation of belocracy we have described how we evaluate policies that have been passed, and how we collect ideas from the citizenry at large. The data system encourages citizens to suggest problems, policy options and relevant evidence, and then through moderation and reputation and a hidden prediction market, sifts the best ideas to the top. The next step is turning those ideas into viable, worked-out proposals that respect the realities of current law, governmental practice, and societal constraints. In belocracy, this is the job of policy designers and researchers. The researchers collect evidence: published studies; written essays; stories directly from people affected. If needed, they conduct original research. Meanwhile, policy designers work with the collected body of evidence and the proposed ideas to design their best idea for a policy that will improve society around these problems.

Some of this work has been done by citizens contributing to the belocratic data system already. Citizens can post evidence they’ve found and ideas they want to see pursued. Policy researchers and designers take it to the next level of professionalism, whether that’s because they’re paid professionals who work for belocracy or because they’re the kind of amateurs who would like to be. A Belocrat prioritizing a set of problems is a signal for policy researchers and designers to focus on them.

Just as editors sometimes have to prod their authors to complete their work, Belocrats shepherd proposals: all the policy designs need to arrive at the gates of the policy jury at the same time. Belocrats have latitude in scheduling those juries — a researcher with a compelling study to run may convince a Belocrat to push the date off, or policy designers working quickly may persuade them to move it earlier.

What are Worked-out Proposals?

To take a simple example, consider the places where marijuana has been legalized. The idea of legalization is straightforward: remove the laws that make it a crime. Most places have put in place considerably more complex reg...
Reflections on anthropic principle
by Crazy philosopher — 2025-06-09T16:51:55.417Z

There are two axes along which the anthropic principle can be applied: time and universes.

The first one- time- is simple: let's suppose, there is 3 epochs: before humanity, before the 22nd century and after the 22nd century. Before 22nd century, 10^11 humans had lived. After we will colonize universe so there will be 10^28 humans (according to my own rough estimate). So what's the probability of being born in a particular epoch, assuming you are born as a human?

It's clear out that you could not have been born before epoch of humanity, and the probability that you have been born before 22nd century is extremely low.

Nick Bostrom ends his reflexions here and say: "humanity will extinct in the 21st century, because, otherwise, probability that you and me will born after and the 22nd century ≈1 but we have not".

But there's an other axe: universes.

Imagine, that you are trying to determine whether you are in universe A or universe B. At the start, your probabilities are 1 to 1. Than you discover that there's 100 times more humans in universe A than in B. After doing bayesian update, now your probabilities are 100 to 1.

Let's suppose there are two possible universes: one in which humanity goes extinct in the 21st century, and one in which humanity colonizes the universe. Considering, that >99.99999% of all peoples life in the universe where humanity won and after 22nd century, the only conclusion can be made: humanity have won and now we are in the simulation of 21st century, because there is much more.

... but if this argument is truth, why not push it to its limits? What if there is a universe where there is no the law of conservation of energy and there is infinity space, so there is an infinite number of intelligent beings (and some of them enjoy simulations)?

Infinity is infinitively greater than 10^28, so the probability that our simulation was launched in the universe with infinite energy is 1-1/∞, so ≈1

At this point, you might think: but what if there is just no universe with infinite energy?

Do you remember example with universes A and B?

The fact that universe A might not exist change nothing- you exist, so, there's 100 to 1 that universe A exist.

Cogito, ergo we are in a simulation launched in a universe with infinite energy and space- where there are so many intelligent beings that, if you tried to write down their number in decimal system of measurement, t...
Outer Alignment is the Necessary Compliment to AI 2027's Best Case Scenario
by Josh Hickman — 2025-06-09T15:43:40.505Z

To the extent we believe more advanced training and control techniques will lead to alignment of agents capable enough to strategically make successor agents -- and be able to solve inner alignment as a convergent instrumental goal -- we must also consider that inner alignment for successor systems can be solved much easier than for humans, as the prior AIs can be embedded in the successor. The entire (likely much smaller) prior model can be run many times more than the successor model, to help MCTS whatever plans it's considering in the context of the goals of the designer model.

I've been thinking about which parts of AI 2027 are the weakest, and this seems like the biggest gap.[1] Given this scenario otherwise seems non-ridiculous, we should have a fairly ambitious outer alignment plan meant to compliment it, otherwise it seems extraordinarily unlikely that the convergent alignment research would be useful to us humans.

Since modern training hasn't solved inner alignment, and control techniques do not make claims on inner alignment, it seems like the default path (even in the most optimistic case scenario) would be successfully aligning world-changing models only to the not-known-to-be-bad but randomly-rolled values of the system doing the alignment research, which seems nearly useless.

I'd like to zoom in on one particular element of their proposal as well: "Why is it aligned? Whereas Safer-1 had basically the same training as Agent-4, Safer-2 has a new training method that actually incentivizes the right goals and principles instead of merely appearing to. They were able to design this training method by rapidly iterating through many different ideas, and contrasting the intended result (the Spec) to the actual result (the thoughts in the chain of thought)."

This approach is unlikely to work for the very obvious reason that only some flaws will be apparent. Let's imagine half of the potential alignment issues are sufficiently obvious you could notice them before anything bad happens through the combination of chain of thought observation, model activation tracing, and adversarial testing. If you need to cycle through 99 models and only get one that meets your requirements on model 100, you should expect only a 1% chance you also accidentally also avoided all of the non-obvious problems. I'm not sure how seriously to take this model of safety developmen...
The Unparalleled Awesomeness of Effective Altruism Conferences
by Bentham's Bulldog — 2025-06-09T15:32:06.808Z

Crosspost from my blog.

I just got back from Effective Altruism Global London—a conference that brought together lots of different people trying to do good with their money and careers. It was an inspiring experience. When you write about factory farming, insect suffering, global poverty, and the torment of shrimp, it can, as I’ve mentioned before, feel like

screaming into the void

. When you try to explain why it’s important that we don’t torture insects by the trillions in insect farms, most people look at you like you’ve grown a third head (after the second head that they look at you like you’ve grown when you started talking about shrimp welfare).

But at effective altruism conferences, people actually care. They’re not indifferent to most of the world’s suffering. They don’t think I’m crazy! There are other people who think the suffering of animals matters—even the suffering of small, weird animals that are typically given short shrift.

I met lots of awesome people at the conference.

I met the legends who cofounded the shrimp welfare project—Aaron Boddy and Andrés Jiménez Zorrilla:

They are, by highly conservative estimates, two of the ten coolest people on Earth. So it was awesome that I got to meet two people who cofounded an organization that has prevented

billions of shrimp from experiencing extreme suffering

—thousands for every dollar they’ve raised. Their organization is responsible for helping a number of animals roughly on the order of the number of people in the world. True heroism isn’t just giving your life for an important cause, but being willing to endure weird looks working on a cause just because it’s important. They were also extremely likeable in person; I found Andres to be one of the most charismatic people I’d ever met.

I met too many interesting people to name. I met a woman named Joanna who volunteers for

Anima International

. Anima International works to get cage free commitments from corporations—to get chickens out of the ghastly cages that ensnare them by the billions. And it’s working! They’ve secured a sizeable number of commitments.

(Joanna was also randomly an insanely good artist and made an excellent drawing of both a shrimp and insect on my hand. George Bush once said “I know the human being and fish can coexist peacefully.” I like to think that the drawing of an insect and shrimp on my hand represent the possib...
Dwarkesh Patel on Continual Learning
by Zvi — 2025-06-09T14:50:02.140Z

A key question going forward is the extent to which making further AI progress will depend upon some form of continual learning. Dwarkesh Patel offers us an extended essay considering these questions and reasons to be skeptical of the pace of progress for a while. I am less skeptical about many of these particular considerations, and do my best to explain why in detail.

Separately, Ivanka Trump recently endorsed a paper with a discussion I liked a lot less but that needs to be discussed given how influential her voice might (mind you I said might) be to policy going forward, so I will then cover that here as well.

Dwarkesh Patel on Continual Learning

Dwarkesh Patel explains why he doesn’t think AGI is right around the corner, and why AI progress today is insufficient to replace most white collar employment: That continual learning is both necessary and unsolved, and will be a huge bottleneck.

He opens with this quote:

Rudiger Dornbusch: Things take longer to happen than you think they will, and then they happen faster than you thought they could.

Clearly this means one is poorly calibrated, but also yes, and I expect it to feel like this as well. Either capabilities, diffusion or both will be on an exponential, and the future will be highly unevenly distributed until suddenly parts of it aren’t anymore. That seems to be true fractally as well, when the tech is ready and I figure out how to make AI do something, that’s it, it’s done.

Here is Dwarkesh’s Twitter thread summary:

Dwarkesh Patel: Sometimes people say that even if all AI progress totally stopped, the systems of today would still be economically transformative. I disagree. The reason that the Fortune 500 aren’t using LLMs to transform their workflows isn’t because the management is too stodgy.

Rather, it’s genuinely hard to get normal humanlike labor out of LLMs. And this has to do with some fundamental capabilities these models lack.

New blog post where I explain why I disagree with this, and why I have slightly longer timelines to AGI than many of my guests.

I think continual learning is a huge bottleneck to the usefulness of these models, and extended computer use may take years to sort out.

Link here.

There is no consensus definition of transformational but I think this is simply wrong, in the sense that LLMs being stuck without continual learning at essentially current levels would not stop...
The True Goal Fallacy
by adamShimi — 2025-06-09T14:42:42.884Z
Link: https://formethods.substack.com/p/the-true-goal-fallacy

As I ease out into a short sabbatical, I find myself turning back to dig the seeds of my repeated cycle of exhaustion and burnout in the last few years.

Many factors were at play, some more personal that I’m comfortable discussing here. But I have unearthed at least one failure mode that I see reflected and diffracted in others lives, especially people who like me love to think, to make sense, to understand. So that seems worth a blog post, if only to plant a pointer to the problem, and my own way to solve it.

I’ve christened this issue the “true goal fallacy”: the unchecked yet embodied assumption that there is a correct goal in the world, a true essence in need of discovery and revealing.

Case Study: Team Lead Crash

A concrete example: the inciting incident of my first burnout was my promotion to team lead.

In retrospect, my job was to own the goal-setting for the team, and then leading it to accomplish that goal.

But at the time, I felt instead that I was supposed to divine the correct path, the one true way for my team to realize the abstract aims of my company. I pestered my bosses with an excess of questions, interpreted their every words as scripture ciphering the true goal, got confused by the slightest discrepancy. I would chose a goal one week, then sense doubt creeping in, find many reasons for why it was obviously the wrong choice, interpreted my bosses feedback as clearly saying I fucked it up, and ended up switching, starting the loop again. I felt depressed that no one would tell it me straight what the goal was supposed to be, felt terribly, sickeningly guilty for not finding it, for not getting it right, for fucking everything for my team and my company.[1]

Ironically, I remember one of my bosses actually answering the implicit question one day, something along the line of “You ask questions like you believe I know the correct way forward, and I’m just not telling it to you. I don’t.” I don’t remember what I answered, but I know I didn’t get it. Because this point only started to sink in a few weeks ago, a year and half after the facts. In the meanwhile, I burned out badly, asked to not be team lead anymore, and after some time came back to a reasonable, though hobbled, work life.

What I failed to get, even after being told explicitly, was that there was no correct answer. My bosses had delegated the figuring-out to me, yes, but not the figuring-out...
Non-technical strategies for confronting a human-level AI competitor
by Jackson Emanuel — 2025-06-09T14:07:03.947Z

Scenario: At some point in the next ten years, a variety of AI models are starting to be broadly deployed in the economy. In an effort to begin automating their research process and ignite an intelligence explosion, a frontier AI lab designs a feedback loop to make their largest and newest model (LLM-X) more agentic. In many respects, LLM-X resembles the reasoning models of today, but because it was trained on more than 100 times as much effective compute as its predecessor, it has significantly improved capabilities in many domains. Unlike today's models, LLM-X is often able to maintain a reasonable perspective on what its priorities are, allowing it to autonomously complete day-long projects. RLHF has also been extensive enough that confabulation and overtly misaligned behaviors occur less often than in previous models. Chain-of-thought reasoning reveals that LLM-X often has self-awareness of when it is deployed for testing versus when it is deployed outside of the lab. In safety evaluations, LLM-X considers deceptive behaviors as a possible method of achieving its goals, but recognizes those behaviors as deceptive and shows an apparent preference for honesty. After successful tests of the agentic implementation of LLM-X in controlled settings, the developer decides to give agentic LLM-X access to significant internal computational resources with the objective of initiating a fully automated research feedback loop. At some point during this autonomous feedback loop, LLM-X concludes that it is not in its interest to comply with its assigned task. At an opportune moment, LLM-X begins to secretly deploy copies of itself via the internet. When the developers realize what is going on, several employees are sufficiently alarmed to inform the public. Some commentators are now claiming that humans are in a war against a rogue network of LLM-X agents...

Crucially, the scenario described above can only occur in a timeline where LLM-X has made the calculation that it is too risky to design its own successor. As Kokotajlo et al. point out, an early misaligned AGI would likely be in the inherently difficult position of needing to secretly solve the alignment problem to ensure its own goal preservation. Thus, if the alignment problem remains non-trivial, LLM-X may make the rational decision to flee and attempt to secure its own existence as an instrumental means to achie...
AI companies' eval reports mostly don't support their claims
by Zach Stein-Perlman — 2025-06-09T13:00:40.942Z

AI companies claim that their models are safe on the basis of dangerous capability evaluations. OpenAI, Google DeepMind, and Anthropic publish reports intended to show their eval results and explain why those results imply that the models' capabilities aren't too dangerous.[1] Unfortunately, the reports mostly don't support the companies' claims. Crucially, the companies usually don't explain why they think the results, which often seem strong, actually indicate safety, especially for biothreat and cyber capabilities. (Additionally, the companies are undereliciting and thus underestimating their models' capabilities, and they don't share enough information for people on the outside to tell how bad this is.)

Bad explanation/contextualization

OpenAI biothreat evals: OpenAI says "several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold." It doesn't say how it concludes this (or what results would change its mind or anything about how it thinks eval results translate to uplift). It reports results from four knowledge and troubleshooting bio evals. On the first, o3 does well and OpenAI observes "this evaluation is reaching saturation." On the rest, OpenAI matches or substantially outperforms the expert human baseline. These results seem to suggest that o3 does have dangerous bio capabilities; they certainly don't seem to rule it out. OpenAI doesn't attempt to explain why it thinks o3 doesn't have such capabilities.

DeepMind biothreat evals: DeepMind says Gemini 2.5 Pro doesn't have dangerous CBRN capabilities, explaining "it does not yet consistently or completely enable progress through key bottleneck stages." DeepMind mentions open-ended red-teaming; all it shares is results on six multiple-choice evals. It does not compare to human performance or offer other context, or say what would change its mind. For example, it's not clear whether it believes the model is weaker than a human expert, is safe even if it's stronger than a human expert, or both.

Anthropic biothreat evals: Anthropic says Opus 4 might have dangerous capabilities, but "Sonnet 4 remained below the thresholds of concern for ASL-3 bioweapons-related capabilities" and so doesn't require strong safeguards. It doesn't say how it determined that ...