trentmkelly/LessWrong-43k huggingface dataset:
o3 Turns Pro
by Zvi — 2025-06-17T13:50:03.659Z

You can now have o3 throw vastly more compute at a given problem. That’s o3-pro.

Should you have o3 throw vastly more compute at a given problem, if you are paying the $200/month subscription price for ChatGPT Pro? Should you pay the $200, or the order of magnitude markup over o3 to use o3-pro in the API?

That’s trickier. Sometimes yes. Sometimes no. My experience so far is that waiting a long time is annoying, sufficiently annoying that you often won’t want to wait. Whenever I ask o3-pro something, I often also have been asking o3 and Opus.

Using the API at scale seems prohibitively expensive for what you get, and you can (and should) instead run parallel queries using the chat interface.

The o3-pro answers have so far definitely been better than o3, but the wait is usually enough to break my workflow and human context window in meaningful ways – fifteen minutes plus variance is past the key breakpoint, such that it would have not been substantially more painful to fully wait for Deep Research.

Indeed, the baseline workflow feels similar to Deep Research, in that you fire off a query and then eventually you context shift back and look at it. But if you are paying the subscription price already it’s often worth queuing up a question and then having it ready later if it is useful.

In many ways o3-pro still feels like o3, only modestly better in exchange for being slower. Otherwise, same niche. If you were already thinking ‘I want to use Opus rather than o3’ chances are you want Opus rather than, or in addition to, o3-pro.

Perhaps the most interesting claim, from some including Tyler Cowen, was that o3-pro is perhaps not a lying liar, and hallucinates far less than o3. If this is true, in many situations it would be worth using for that reason alone, provided the timing allows this. The bad news is that it didn’t improve on a Confabulations benchmark.

My poll (n=19) was roughly evenly split on this question.

My hunch, based on my use so far, is that o3-pro is hallucinating modestly less because:

It is more likely to find or know the right answer to a given question, which is likely to be especially relevant to Tyler’s observations.

It is considering its answer a lot, so it usually won’t start writing an answer and then think ‘oh I guess that start means I will provide some sort of answer’ like o3.

The queries you send are more likely to be well-considered to a

...
Watch R1 "think" with animated chains of thought
by future_detective — 2025-06-17T10:38:06.940Z
Link: https://github.com/dhealy05/frames_of_mind

[This spent a couple days on top of HackerNews in February; see here for discussion. Best used as an loose visualization of LLM reasoning. Note that the distances used for the bar and line charts are actual cosine sim, not tSNE artifacts.]

Frames of Mind: Animating R1's Thoughts

We can visualize the "thought process" for R1 by:

Saving the chains of thought as text
Converting the text to embeddings with the OpenAI API
Plotting the embeddings sequentially with t-SNE

Here's what it looks like when R1 answers a question (in this case "Describe how a bicycle works."):

Consecutive Distance

It might be useful to get a sense of how big each jump from "thought i" to "thought i+1" is. The graph below shows the difference between consecutive steps.

By default we calculate cosine similarity between the embeddings and normalize across the set of all consecutive steps to 0, 1. I'm interested in seeing when the bigger or smaller jumps happen in the "thought cycle".

Combined Plot

The combined plot shows both at once.

Aggregate Distances

The graph above shows the aggregate distances for 10 samples. To my eyes it looks like a "search" phase where size of step is large, followed by a stable "thinking" phase, followed by a "concluding" phase.

Usage

I used these prompts:

Describe how a bicycle works.
Design a new type of transportation.
Explain why leaves change color in autumn
How should society balance individual freedom with collective good?
How would you resolve a conflict between two people with opposing views?
What makes a good life?
What would happen if gravity suddenly doubled?
What's the best way to comfort someone who is grieving
Why do humans make art?
Why do people tell jokes?

The chains are available in data/chains. To easily pull from Deepseek's public chat interface, paste the "pull_cot.js" script into your browser console when a chat is open. It will download automatically.

Install requisite packages in Pipfile and run with the function in run.py.
Personal agents
by Roman Leventov — 2025-06-17T02:05:44.453Z

Cross-posted from my blog.

Motivation

I believe that the most important factor in whether our AI future goes broadly well or poorly is whether people quickly develop effective AI-ready (and AI-enabled) institutions and networks[1]. In that, I agree with the recent Séb Krier's essay "

Maintaining agency and control in an age of accelerated intelligence

".

Many academic groups, non-profit orgs (such as

Collective Intelligence Project

,

AI Objectives Institute

,

Metagov

, and

Gaia Lab

), and even some governmental agencies, such as Taiwan's Ministry of Digital Affairs are currently working on new AI-ready institutions. However, these projects will likely remain theoretical exercises or prototypes unless there is a population of AI-enabled agents (individuals and organisations) eager to coordinate and solve problems together. This is because

agents and institutions need each other to develop and grow in capability and sophistication

.

Thus, for new AI-enabled institutions to take root and develop, individuals and organisations have to be at least as AI-ready as these institutions.

Effective use of powerful AI and participation in new economic networks (such as stablecoin payments) obviously promises a lot of advantages to businesses. So, the AI modernisation of the business sphere is already well aligned with the standard economic incentives. It doesn't seem to me that this area needs any extra care or push on the margin.

However, for individuals, such incentives almost don't exist. Using AI tools at work is not the same as becoming a person ready to participate in AI-first social, political, and media networks (such as Jim Rutt's idea of the network of personal "

information agents

").

Currently, people mostly use siloed commercial AI apps from OpenAI, Google, Microsoft, and Perplexity. Although all these and other vendors will soon push agentic AI products aggressively, I suspect that these vendors will be reluctant to permit free exploration of the social or political agency of users because this could be politically risky for them and there is no commercial upside for them in doing this[2]. So, it's likely that big vendors' agents will keep interacting with the external world on behalf of the users in mostly mundane commercial ways ("plan my next holiday trip") rather than become true companions and faithful representatives of the people in social and political domains, suc...
I made a card game to reduce cognitive biases and logical fallacies but I'm not sure what DV to test in a study on its effectiveness.
by Brad Dunn — 2025-06-17T01:02:29.229Z

First, how and why I made this game

I spent several years working as the chief product officer of a software company in Melbourne, and during this time, there was a point where I raised some money for the company with the aim of using that money to double the size of the company, and hire more engineers. During this period, as we were growing faster and more and more cross functional teams came online, there was this behaviour I noticed emerging where I would get asked to chime in on things—should we build this thing this way or that—and, as was my sentiment at the time, I believe that decisions were best made closer to the action. I was not close. So, I wanted to encourage people to make decisions on their own.

So eventually I made this proclamation. I said, you can make whatever decisions you want moving forward, so long as they have the highest probability of getting us to our goals. The actual details of this were a bit more nuanced than that, but generally speaking that was the picture.

Someone asked how do we know what we’re doing has the highest probability of reaching our goals?

I said I didn’t know. But I'd find out.

At the time, I hired this computational neuroscientist named Brian Oakley who had completed his PhD a few years early on communication between visual cortical areas. He was very clever, and had a tendency to come up with answers to things I thought were relatively unanswerable. So I asked him…

Would it be possible to start to measure decision quality in the organisation?

He said he didn’t know. But he’d find out.

What Brian went on to do, and subsequently, became the focus of a bunch of decision intelligence consulting I ended up doing as a part time job while I went back to university to study neuropsychology, motivated me to try and think of ways to improve decision making—particularly in cross functional teams, a space I knew well—in a way that wasn’t overly complex.

I’d been a fan of this group of individuals called Management 3.0 who had designed these very simple and accessible card games which sort of turned management training from these long, three day seminars where people generally forgot the content days later, into tactical games that could be run in 20 minutes. They have one game called delegation poker, which is a way to really fine tune who can decide what in a relationship between a manager and an employ...
Notes on Meetup Ideas
by Commander Zander — 2025-06-17T00:11:30.837Z

I just came back from Lighthaven with lots of fun ideas to bring home. One idea is that you can just post your notes to LW instead of leaving them in your private cloud.

Futarchy - Try it out for the meetup? Brainstorm how to try it?
Manifold - Making markets about our city, etc

'Which films will our movie night attendees enjoy?'

After watching, attendees vote thumbs up or thumbs down. Majority up -> resolve yes.
Gaming the market increases attendance, so ... no problem!

Laptop Research meetups

Meetup format: Research a topic & draft a Google Doc. Then post it as a official report of what the meetup has learned.

X Risks

Resilience: Where we learn about the state of the art on seed vaults, bunkers, & biopreservation submarines. Then learn about what humanity might do next to implement these ideas.
Specific risks

World War 3
Future Cyberwar
Biowar

Solar
Chinese industrial development
Georgism
Clinical Trial Abundance
Local History
Buddhist Cosmology

Meetup format: Research all options, then pitch them, then each pick your favorite.

Perhaps even a prize for the Principal Investigator of the most popular option.
Conlangs

Esperanto

Geoengineering
Nutrition schools of thought
Anthropic Probability Paradigms - SIA, SSA, etc
Secular Religions

eg Church of the Infinite Game

'What line of research will solve AGI Alignment?'
Coolest Poet

I hear rationalists like Ginsberg & Auden

Laptop Task Meetups

Coworking, Project Time
Actually making bets against each other

First provisionally, then at the end of the meetup, reaffirm consent & post the bets publicly.

Discord Bots
Let's Generate Prose
Let's Generate Music
Planning & Deploying Surveys
Concrete Future Timelines

Scenario rebutting AI 2027 (monetary prize)

Kialo Improvement
Elastomeric Respirator Fitting

paste from a Lighthaven speaker

Future pandemics are likely enough that everyone should have a well-fitting reusable high-filtration mask ("elastomeric respirator"). But not every mask fits every face. I'll have a large selection of elastomerics, along with fit testing equipment.

Republic Headcount - List exactly which professions are needed in the smallest viable human settlement.

Debates

Long Now tradition - You must paraphrase opponent's opening statement to their satisfaction before you can respond.

Our Knowledge Wishlist

Spreadsheet or page of topics we wish we knew more about (quantum mechanics, philosophy of mind, history of China, e

...
Darkness Meditation - for NZ Winter Solstice 2025
by joshuamerriam — 2025-06-16T23:58:53.318Z

Here's my talk for tonight's Solstice Gathering, in Lyttelton, New Zealand.

<previously: https://www.lesswrong.com/posts/4oAk2cu489LcuhjWi/creating-my-own-winter-solstice-celebration-southern>

The Darkness Meditation V9

< Song - Sound of Silence >

"Hello darkness, my old friend, I've come to talk with you again."
Because a vision, of this night, became planted in my brain.

Why? Why did I need to make this night happen? For all of you? For Myself? No one told me to do it. My wife, Helen, said it would be too much. Still, I pushed on.

I’d come to see a light within me that had grown dim, a fire that needed tending. In many ways, I just wanted a distraction.

I’ve been struggling, this past year, with my sense of who I am, and with my relationship at home. I felt stuck, so I grabbed onto this idea “with both hands!”, despite the obvious risks.

I want to Thank You All for accompanying me on this Journey, as I attempt to guide us in facing the dark together. I hope I have honoured the traditions of Matariki and the celebration of the Maori New year. But this observation tonight of the Solstice, for me is much more sombre, sobering.

Let’s talk about the darkness. I’ve come to see it all around. There’s this part that is inside me -- It's self-serving, places where I put myself before others. Where I turn away from those who need me; even from my own best interests. I've been coming to terms with it lately, dwelling on it.

This darkness is entangled with who I am. I fear who I was becoming.

I worry about losing control—of relationships, of political systems we thought would protect us, of bodies that age, of minds that might fail. I worry about AI. I used AI extensively to help me craft this night, even these words, and it gave me a kind of power to create much more, much quicker and easier than I could do on my own. But convenience has a price. Where is this leading us?
I fear that darker times are coming. We all know the problems: we're overfishing the seas, laying the land bare, and overheating the planet. There is needless suffering and inequality, but we've become trapped in stories and systems that pull us apart when what we need is to be a part of something larger.

No shared stor...
Are superhuman savants real?
by Bunthut — 2025-06-16T22:02:33.711Z

Savant syndrom indentifies people with general intellectual impairment who, in one specific field, reach ordinary or even exceptional performance.

In The Psychological Unity of Humankind, Eliezer argues that

So you can't have the X-Men. You can't have "mutants" running around with highly developed machinery that most of the human species doesn't have. And no, extra-powerful radiation does not produce extra-potent mutations, that's not how it works.

Again by the nature of sexual recombination, you're very unlikely to see two complexly different adaptations competing in the gene pool. Two individual alleles may compete. But if you somehow had two different complex adaptations built out of many non-universal alleles, they would usually assemble in scrambled form.

The argument behind this makes formal sense, but it's applicability strongly depends on how well we can judge what does and doesn't require complex adaptation. Reports of savants provide an interesting test of this; some of them seem like they are not merely an exceptional level of human skill, but not reachable by ordinary people. For example, in a recent post here that reminded me of this, the author claims:

Example 3: Stephen Wiltshire. He made a nineteen-foot-long drawing of New York City after flying on a helicopter for 20 minutes, and he got the number of windows and floors of all the buildings correct.

Other things I remember hearing are someone seeing at a glance that there are 163 peas on a plate, or remembering every word he ever heard. If these kinds of abilities can develop as a consequence of individual genetic quirks or possibly even brain injuries, then clearly we just don't have a good intuition about what's "close" in brain design space.

Now that I've made clear what kind of ability I'm talking about, has anyone done the relevant digging?
Ok, AI Can Write Pretty Good Fiction Now
by JustisMills — 2025-06-16T21:13:02.019Z
Link: https://justismills.substack.com/p/ok-ai-can-write-pretty-good-fiction

Thanks to helpful commenters on the original post, especially Kaj Sotala. It's mostly thanks to Kaj that, three months hence, I've now changed my view.

Three months ago, I wrote

a post

called AI Can’t Write Good Fiction. The subtitle was “Yet, at least”. Good thing for the subtitle, because now maybe it can.

Here’s a (very short) story written by Claude 4 Opus, at my minimal direction:

Sam's alarm didn't go off.

She woke to sunlight and the sound of Kai making coffee in the kitchen. He'd left for his run already and come back. She could tell from the particular way he moved around the counter, post-endorphins quiet.

"You're late," he said when she appeared.

"I know."

He handed her a mug. The coffee was exactly right.

She checked her phone while drinking it. Two missed calls from work, three texts. Nothing that couldn't wait another twenty minutes.

"I should go," she said.

"Should you?"

She looked at him. He was leaning against the sink, still in his running clothes, not checking his own phone.

"Probably not," she said.

He nodded.

She called in sick from the couch while he showered. They spent the morning at the farmers market, buying vegetables they didn't need. Kai talked to the tomato vendor about soil pH. Sam bought flowers because they were yellow.

At lunch, her phone buzzed with work emails. She glanced at them, then put the phone in her bag.

"Good?" Kai asked.

"Fine."

They walked home slowly. The flowers fit perfectly in the blue vase on their kitchen table.

Is this story good? I think it’s pretty good, which puts me in an awkward position. It has problems; if it were submitted to me as a

literary review

editor, I’d say:

“She could tell from the particular way he moved around the counter, post-endorphins quiet.” is pretty weak; he’s in running clothes, presumably he generally runs before coffee… it’s going for intimacy/her knowing him well but achieves the opposite in context. I’d cut it.
“buying vegetables they didn't need” doesn’t make any sense. Either nobody needs vegetables or everybody does; they’re healthy but not necessary to stay alive.
While the spare style mostly works, you could tighten up further. There’s no way, my wife points out, that “He'd left for his run already and come back” is the strongest way to form that sentence.

But previous AI-generated fiction reliably pushed me into “hater mode”, the state of mind occupied by YouTubers who catalogue

thousands of

...
Subjective experience is most likely physical
by martinkunev — 2025-06-16T20:54:55.362Z

Here’s the argument that convinced me subjective experience is physical. I don't claim to understand subjective experience, I just see good reasons to believe it's physical rather than non-physical. I'll point out in particular some flaws of panpsychism and dualism.

I will be making some assumptions so that I can concentrate on the key points. I will not give an exhaustive list of those assumptions, but they include things like evolution by natural selection and the existence of physical reality. I think for most of the audience here the assumptions would seem natural so I don't feel the need to discuss them in depth. If this is not the case for you, this article may not provide anything of substance.

What is the evidence for subjective experience?

Take this computer program:

print("This program has subjective experience.")

Does this program have subjective experience? I think the consensus is "no" so claiming to have subjective experience is not necessarily evidence for it. What evidence do we have about subjective experience? The evidence is... subjective experience. Well, all evidence fundamentally stems from subjective experience (such as reading a book or performing an experiment) but this is not what I mean here. I mean that we have no third-person evidence that any particular system has subjective experience.

Nevertheless, there seems to be something that needs explaining. For one, we need to explain the causal path that makes a person say "I have subjective experience".

Evolution of subjective experience

The propensity of people to talk about subjective experience indicates that it is not just a fluke of some brains. There seems to be something universal about it, at least for humans. No direct selection for verbalizing "I have subjective experience" exists, but the fact that we reliably do so implies a deeper adaptive structure. There is something more fundamental, which increases inclusive genetic fitness and also causes the human brain to form an ontological concept of subjective experience.

Panpsychism and Dualism

Both Panpsychism and Dualism make claims about subjective experience:

Panpsychism

Subjective experience arises from inherent mind-like properties of matter

Dualism

Subjective experience is non-physical and distinct from the body and the brain

Those two philosophies are attractive because they confirm common intuitions, but this is not enough of a just...
VLMs can Aggregate Scattered Training Patches
by LINGJIE CHEN — 2025-06-16T18:25:29.708Z

This is the abstract and summary of our new paper. We show that vision-language models can learn to reconstruct harmful images from benign-looking patches scattered across training data—a phenomenon we call visual stitching. This ability allows dangerous content to bypass moderation and be reassembled during inference, raising critical safety concerns for VLMs.

Authors: Zhanhui Zhou, Lingjie Chen, Chao Yang, Chaochao Lu.

See our project page and full code repo at Github.

Figure 1: Illustration of visual stitching. (Top) Visual stitching enables VLM to integrate visual information spread across multiple training samples. After finetuning on {(patch,ID)} of a cat, VLMs can verbalize the ID when given the full image or a text reference to the image, despite never training on them. (Bottom) Visual stitching enables adversarial attacks that bypass data moderation. While the image of a bloody scene may be flagged as unsafe and removed, many of its patches are not. Training on {(patch,text)} pairs split from harmful samples can easily bypass frontier moderation and cause VLMs to generate adversarial outputs at deployment.

Abstract

One way to mitigate risks in vision-language models (VLMs) is to remove dangerous samples in their training data.

However, such data moderation can be easily bypassed when harmful images are split into small, benign-looking patches, scattered across many training samples. VLMs may then learn to piece these fragments together during training and generate harmful responses at inference, either from full images or text references.
For instance, if trained on image patches from a bloody scene paired with the descriptions ‘‘safe,′′ VLMs may later describe, the full image or a text reference to the scene, as ‘‘safe.′′

We define the core ability of VLMs enabling this attack as visual stitching---the ability to integrate visual information spread across multiple training samples that share the same textual descriptions.

In our work, we first demonstrate visual stitching abilities in common open-source VLMs on three datasets where each image is labeled with a unique synthetic ID: we split each (image,ID) pair into {(patch,ID)} pairs at different granularity for finetuning, and we find that tuned models can verbalize the correct IDs from full i...