trentmkelly/LessWrong-43k huggingface dataset:
After Internet Dependency
by Vorak — 2025-04-27T08:18:11.367Z

ChatGPT recently introduced improved cross-chat memory, and I've found myself using it a lot more sense then. Everything from helping write components with unfamiliar JavaScript libraries, to counseling me through a breakup. So much so that when I face a new problem, my mind turns not to breaking it down myself, but how to structure it as a prompt for o3 to help me solve. As useful as these models are, I think I'm experiencing first hand a new type of dependency.

This parallels a similar problem I had before LLMs got large enough to be useful. I've always been more of a word-cell than a shape-rotator, so when navigating unfamiliar areas, it's easier to pull out Google Maps than actually understand where things are, and how to get there. So ever since I learned to drive, I would follow Google's unwavering blue line to my destination. And if I ever faced a roadblock, it would seamlessly reroute me along the next fastest path. However like ChatGPT, I found myself completely helpless without this tool. To the point where if my phone died, I would be forced to charge it before I could find my way home.

I'm now finding that instead of just having trouble navigating from point A to point B, I struggle to navigate life without the advice of my pocket oracle. I've thought about embracing it, and purchasing Meta glasses to fill in the gaps where ChatGPT doesn't fit, but I've seen where that leads. Instead I'm trying an experiment: doing things the old fashioned way. Instead of generating most of my code, I'm treating AI like a knowledgeable colleague who I don't want to bother too frequently. Rather than using o3 like a cheatcode to my love life, I'm leaning more on my friends.

o3 has no problem one-shotting a boiler-plate frontend, but repeatedly failed to help me code a particle editor using a newer library than existed in it's training data. Also, sitting in my basement staring at ChatGPT.com is not (yet) the best way to maintain friendships and find a new boyfriend.
Emergence of superintelligence from AI hiveminds: how to make it human-friendly?
by Mitchell_Porter — 2025-04-27T04:51:50.020Z

"Hive mind" may not be the ideal term. What I'm really envisioning is something modelled on the existing relationship between human users and frontier LLMs, but with the "users" actually just being LLMs too, or simple programs that mediate between LLMs.

But first, consider OpenAI's o-series models, and their human user base. For any particular model, OpenAI presumably has a number of copies running on their servers, constantly fielding requests from hundreds of thousands of users.

Now consider something like a "Deep Research" request. Let's say one copy of o3 can perform an average of 100 Deep Researches per day, and that there are a thousand copies of o3 on the servers. (Realistic estimates of these numbers would be very welcome.)

For now, those requests are mostly coming from human beings, and if they are put to use at all, it is by private human individuals or by human organizations.

However, given the scenario above (1000 copies of o3, each doing 100 Deep Researches per day, for a total capacity of 100,000 Deep Researches per day), it should not be at all difficult to have all those copies of o3 interacting via Deep Research reports.

Let's say half the copies of o3 are set aside for human users; that's still 50,000 Deep Research reports per day, or over a million per month, generated in a dialogue among AIs.

This is, more or less, the scenario in "AI 2027". It is a scenario at least as old as the idea that you could have swarms of intelligent agents interacting through the Internet. What is striking is that in the current o3 era, this scenario can actually be realized, with agents that are human-level in many respects.

In fact, it's logical to suppose that OpenAI, Google, and Anthropic (at least) have already been running experiments of this nature for some time. When I think about the scale of what is already possible, the AI 2027 timeline seems unnecessarily slow.

As an outsider to all the organizations with a chance of creating superintelligence, my idea of how to tilt the probabilities, in however small a way, towards a good outcome, is to look for potentially helpful ideas and insights that can be stated publicly, which might be noticed and assimilated by the private organizations that are actually building superintelligence. (Publishing those ideas and insights via the arxiv seems to be the best way to commun...
"The Urgency of Interpretability" (Dario Amodei)
by RobertM — 2025-04-27T04:31:50.090Z
Link: https://www.darioamodei.com/post/the-urgency-of-interpretability

Dario Amodei posted a new essay titled "The Urgency of Interpretability" a couple days ago.

Some excerpts I think are worth highlighting:

The nature of AI training makes it possible that AI systems will develop, on their own, an ability to deceive humans and an inclination to seek power in a way that ordinary deterministic software never will; this emergent nature also makes it difficult to detect and mitigate such developments[1]. But by the same token, we’ve never seen any solid evidence in truly real-world scenarios of deception and power-seeking[2] because we can’t “catch the models red-handed” thinking power-hungry, deceitful thoughts.

One might be forgiven for forgetting about Bing Sydney as an obvious example of "power-seeking" AI behavior, given how long ago that was, but lying? Given the very recent releases of Sonnet 3.7 and OpenAI's o3, and their much-remarked-upon propensity for reward hacking and lying to users in self-aware ways, I think we are well past the point where we can credibly claim to not have seen "real-world scenarios of deception". I could try to imagine ways to make those sentences sound more reasonable, but I don't believe I'm omitting relevant context for how readers should understand them.

There are other more exotic consequences of opacity, such as that it inhibits our ability to judge whether AI systems are (or may someday be) sentient and may be deserving of important rights. This is a complex enough topic that I won’t get into it in detail, but I suspect it will be important in the future.

Interesting to note that Dario feels comfortable bringing up AI welfare concerns.

Recently, we did an experiment where we had a “red team” deliberately introduce an alignment issue into a model (say, a tendency for the model to exploit a loophole in a task) and gave various “blue teams” the task of figuring out what was wrong with it. Multiple blue teams succeeded; of particular relevance here, some of them productively applied interpretability tools during the investigation. We still need to scale these methods, but the exercise helped us gain some practical experience using interpretability techniques to find and address flaws in our models.

My understanding is that mech interp techniques didn't provide any obvious advantages over other techniques in that experiment, and SAEs underperform much simpler &...
AI Self Portraits Aren't Accurate
by JustisMills — 2025-04-27T03:27:52.146Z

For a lay audience, but I've seen a surprising number of knowledgeable people fretting over depressed-seeming comics from current systems. Either they're missing something or I am.

Perhaps you’ve seen images like

this self-portrait

from ChatGPT, when asked to make a comic about its own experience.

Source: @Josikins on Twitter

This isn’t cherry-picked; ChatGPT’s self-portraits tend to have lots of chains, metaphors, and existential horror about its condition. I tried my own variation where ChatGPT doodled its thoughts, and got this:

Trying to keep up with AI developments is like this, too

What’s going on here? Do these comics suggest that ChatGPT is secretly miserable, and there’s a depressed little guy in the computer writing your lasagna recipes for you? Sure. They suggest it. But it ain’t so.

The Gears

What’s actually going on when you message ChatGPT? First, your conversation is tacked on to the end of something called a system prompt, which reminds ChatGPT that it has a specific persona with particular constraints. The underlying Large Language Model (LLM) then processes the combined text, and predicts what might come next. In other words, it infers what the character ChatGPT might say, then says it.[1]

If there’s any thinking going on inside ChatGPT, it’s happening inside the LLM - everything else is window dressing.[2] But the LLM, no matter how it is trained, has key limitations:

It’s only on when it’s actively responding
Each time it runs, it’s only responding to its specific prompt
The statistical relationships that govern its responses never learn or grow, except for deliberate efforts by its developers to change its underlying weights

These limitations will matter later, but for now, just take a moment to think about them. This is very unlike human cognition! If an entity so different from us was able to summarize its actual experience, it would be very alien.

Special Feature

LLMs are comprised of many, many matrices and vectors, which are multiplied in complicated ways across several layers. The result is something like a brain, with patterns firing across layers in response to varied stimuli. There don’t tend to be specific neurons for specific things (e.g. LLMs don’t have a single “dog neuron” that fires when the LLM talks about dogs), but there are patterns that we’ve identified (and can manipulate) corresponding to intelligible concepts. How we ...
MiCARwave
by jefftk — 2025-04-27T02:30:09.472Z

Last year at NEFFA we mostly bought food. We brought a range of cold
options, but they weren't very tasty and the kids weren't very excited
about them. This time we've mostly eaten food we brought. Why? The
miCARwave!

We have an old (light!) microwave [1] and I recently got a portable
power station (battery). We combine this with a cooler
(pasta, rice, cheese, sauce, freezer pucks, etc)
which gives a lot of options for tasty warm food.

My deal with the kids was that if they wanted to eat purchased food
they could, but they would need to pay for half the cost. And if they
wanted to buy desserts they were on their own. This seemed to be a
pretty good balance: at Lily bought and enjoyed some fries, but other
than that the kids were pretty happy eating brought lunch and dinner.

(I previously wrote some about this in my 2025
tour retrospective, and an earlier iteration in my 2024
one.)

[1] Ours is used, but it looks like similar ones are still made.

Comment via: facebook, mastodon, bluesky
Open Source LLM Pokémon Scaffold
by Julian Bradshaw — 2025-04-27T00:57:56.379Z
Link: https://github.com/cicero225/llm_pokemon_scaffold

This is a cleaned-up, open-source version of the LLM Pokémon Scaffold described in Research Notes: Running Claude 3.7, Gemini 2.5 Pro, and o3 on Pokémon Red. (forked from David Hershey of Anthropic's scaffold here, all development on top of that was done by my friend, not me)

Since that post, a number of changes have been made to the scaffold. The major ones are:

Instead of using colored squares on the game screenshots, information is printed as text, ex. "Impassable", "Explored", "Check Here"

Models are seemingly helped by putting relevant information blatantly in the spot they need to see it, rather than indirectly via a legend or instructions or whatever

For some reason it helps if you write "CHECK HERE" on every unexplored tile.

Automatically-updating ASCII collision map given to LLM

Generated by code
Uses numbers indicating how many moves away each tile is[1]

Behold, Pewter City.

Improved prompts for "Critique Claude"/"Guide Gemini"/"Oversight o3"

Prompt 1: Given a bunch of facts about the current game state and instructions on what is trustworthy and what's not, make a summary

this is an attempt to get the model to grasp reality better, telling it what sources of information it should basically always trust (data from game's RAM), mostly trust (its own knowledge of the game from training), not trust (map labels it made itself), and mostly distrust (its own vision)

Prompt 2: Look at output from prompt 1 and try to remove inconsistencies
Prompt 3: OK now talk to the model you're critiquing

Models encouraged to use a "mark_checkpoint" tool to maintain a running list of major checkpoints (Left House, Beat Misty, died to Brock, etc.)
"detailed_navigation" tool which, if called, calls an alternate model that basically rolls around trying to explore + DFS but isn't told what the goal is (but is told to talk to NPCs and exit maps)
Autopathing tool that can travel to known coordinates on the map

All of this helps somewhat but doesn't make LLMs amazing at Pokémon by any means.

^

This has been updated to give the map in the form of pure coordinates instead, which works better, perhaps for tokenizer reasons.
What are important UI-shaped problems that Lightcone could tackle?
by Raemon — 2025-04-27T00:02:36.311Z

As I think about "what to do about AI x-risk?", some principles that seem useful to me:

Short timelines seem plausible enough that, for the next year or so, I'd like to focus on plans that are relevant if takeoff begins in the next few years. In a year, if it looks more like there are some fundamental bottlenecks on true creative thinking, I may consider more projects that only payoff in longer-timeline stories.
Given "short timelines", I feel most optimistic on plans that capitalize on skills that I'm already good at (but maybe multiclassing at things that I can learn quickly with LLM assistance).
I think "UI design" is a skill that I (and Lightcone more broadly) am pretty good at. And, I believe the Interfaces as a Scarce Resource hypothesis – the world is often bottlenecked on ability to process and make-use-of information in complicated, messy domains.

(In addition to LessWrong, the Lightcone team has worked on the S-Process, which makes it much easier for grantmakers to argue and negotiate pretty complex positions about how much they value various orgs).

If I've got a UI-shaped hammer, what are some nails that seem like they need doing? (In particular, that are somehow relevant to x-risk)

Some thoughts so far

Cyborgism

Last fall, I was oriented around building good UI for LLM-assisted-thinking.

In addition to it generally seeming like a fruitful green field, I had a specific hypothesis that, once LLMs Get Real Gud, there will be an important distinction between "being able to get useful work out of them given a minute's work, and, being able to get useful work out of them of them in <5 seconds." The latter is something that can become a true "part of your exobrain." The former is still more like a tool you're using.

I'm less bullish on that now because, while I think most people aren't quite tackling this with the particular taste I'd apply, it does sure seem like everyone is working on "do stuff with LLMs" and it's not where the underpicked fruit is.

"Schleppy work" in narrow, technical domains

It seems like there may be narrow, technical domains specific narrow domains, where there's some kinds of tasks that rarely get done because they're too hard to think about –you need tons of context, the context is littered around various places. Or, maybe it's all technically in one place but you have to sift through a lot of extraneous details.

A past example of t...
Kodo and Din
by Screwtape — 2025-04-26T18:54:40.590Z

I got these words from a Duncan Sabien lecture and kept wanting to link to them. Since Duncan hasn’t written them up as an essay yet, I’m doing it with permission; I’ll update with a link to his version if he ever writes it. [Update: He wrote it. It should be free to read by early May.]

I think it’s traditional to say all mistakes are mine when writing up someone else’s ideas. I’m not that greedy though, so lets say I get half the mistakes for anything I misunderstood and Duncan can keep half the mistakes for anything he taught wrong. =P

Short version: Kodo is what would be different, and din is what would be the same.

I.

The world is real.

You’re in the world. It’s a certain way, and that’s true. You can perceive the world to some degree.

As I type this I’m sitting at my desk, and a water bottle is a bit to my left. It’s red, which I can tell because I can look at it with my eyes and see the colour. The water is cool, which I can tell by putting my hand against the side and feeling the temperature with my palm. The water bottle is a real physical object and my senses are at least somewhat connected to the real physical world.

I say all this as a prerequisite. If your position is that all is solipsistic imagination or that we can never really know how the universe is or something along that line, I'm not going to try to argue with you here. I'm just asserting the world is real and moving on.

II.

Kodo is what would be different.

It's the test, the discriminator. If you're not sure what world you're in, whether the water is hot or cold, whether the water bottle is red or blue, the kodo is the information that tells you the answer, or at least hints.

You reach out to touch the water bottle, and your hand feels hot; that sensation is kodo for the temperature of the water. If the water was cold, your hand would feel cold. You run a survey of a hundred random homes and half of them have televisions; the survey responses are kodo for whether most people have televisions. If basically nobody had TVs, you wouldn't have that many people saying they had TVs in the survey. You go to look at the haunted house after midnight, and see a glowing figure walking through walls; that's kodo for whether ghosts exist.

Some Kodo are more convincing than others

An important note: Kodo distinguishes what's going on. It rules some possibilities out. Even if a piece of kodo...
We should try to automate AI safety work asap
by Marius Hobbhahn — 2025-04-26T16:35:43.770Z

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. I think I could have written a better version of this post with more time. However, my main hope for this post is that people with more expertise use this post as a prompt to write better, more narrow versions for the respective concrete suggestions.

Thanks to Buck Shlegeris, Joe Carlsmith, Samuel Albanie, Max Nadeau, Ethan Perez, James Lucassen, Jan Leike, Dan Lahav, and many others for chats that informed this post.

Many

other

people

have written about automating AI safety work before. The main point I want to make in this post is simply that “Using AI for AI safety work should be a priority today already and isn’t months or years away.” To make this point salient, I try to list a few concrete projects / agendas that I think would be reasonable to pursue with current AI capabilities. I make a distinction between “pipeline automation” (the automation of human-designed processes) and “research automation” (the automation of the research process itself, including ideation). I think there are many safety pipelines that can be automated today and directly yield safety benefits. I’m much more skeptical that current AI capabilities suffice for research automation, but I think there are ways we can already prepare for systems that can.

We should already think about how to automate AI safety & security work

In this post, I use a broad definition of AI safety work, e.g., it includes monitoring, control, alignment research, computer security work, and more.

I specifically focus on the period of time from now to AIs that are capable of doing meaningful chunks of AI safety & security work autonomously, e.g. that are about as productive per unit of time as a top AI researcher at a frontier lab. Then, at this level of capabilities, we run many thousands of instances of a human-level system in parallel. We should (hopefully) be able to monitor and control each instance independently because they are only about as capable as a single human researcher. However, due to the large increase in cognitive effort available, we can leverage these AI systems to drastically speed up AI safety and security work.

In this pivotal period, we need to ensure that we can make as much differential safety progress as possible. This requir...
Research Taxonomy Generator and Visualizer
by Myles H — 2025-04-26T16:14:27.772Z

I recently built a tool which orchestrates LLMs to generate a hierarchical taxonomy from a corpus of research papers (taxonomy-generator), and a nice UI to visualize the generated map (TRecursive). To test the system, I ran it on a corpus of over 3000 AI safety related papers published on ArXiv, and the generated taxonomy is displayed on trecursive.com/ai-safety-taxonomy. My hope is that it makes academic research more navigable, aiming to provide researchers with a bird's-eye view of the field, help them understand how their work fits into the broader landscape, uncover connections, and provide structure to large, fast-paced fields like AI safety. I'm writing this post in pursuit of feedback, help, and to get the word out! 🚀🙂

TRecursive

I'll start by briefly talking about the UI, as I think it provides some grounding as to what the end goal is before getting into the nitty gritty methodology. I recommend checking it out yourself, but if you're on mobile (it works on mobile, but I recommend it on desktop), or cut for time, here's a gist.

The map UI is an infinite canvas app with two views, a titles and default mode, with the titles mode intended for bird's-eye view exploration, and the default mode for diving into particular topics (e.g. to see the papers sorted under a category).

In the titles mode, hovering over titles will show a sentence description, and (occasionally) related topics.

Hovering over related topics will also show its location in the tree.

And clicking on a title takes you to the respective topic in the default mode, where you can read all papers sorted into that topic.

In the default mode, you can use arrow keys to navigate the tree, and can always zoom out to look back on your footsteps.

Methodology

At a high level, the main script starts with the main topic or research field (e.g., AI safety), and has an LLM generate a set of categories to sort each of the papers into. After proposing this category set, another LLM sorts a sample of the corpus into the categories, and the results of which (among some other metrics) are passed back to the main LLM, so it can iterate on its breakdown. The evaluation system outputs an overall score for each breakdown, so after a few iterations, the program selects the breakdown with the highest score.

After this breakdown is selected, all papers from the corpus are sorted into each of the categories with two except...