trentmkelly/LessWrong-43k huggingface dataset:
Large Language Models suffer from Anterograde Amnesia
by Annapurna — 2025-06-06T01:30:18.014Z
Link: https://jorgevelez.substack.com/p/memento

Memento (2000)

My wife and I are power users of Large Language Models (LLMs). My go-to LLM has been Google Gemini, while she has been extensively using ChatGPT for almost a year. We make an effort to try these models for both personal and professional tasks, making our lives more efficient.

As power users of LLMs, we constantly run into their shortcomings. One of the most obvious shortcomings is the models' poor short-term memory. This shortcoming also exists in human beings via a disorder called anterograde amnesia.

What is Anterograde Amnesia?

Anterograde amnesia is a type of memory loss that occurs when you can’t form new memories. In the most extreme cases, this means you permanently lose the ability to learn or retain any new information.

Temporary anterograde amnesia often occurs with binge drinking. If you’ve ever been blackout drunk, you have experienced anterograde amnesia. Permanent anterograde amnesia is quite rare, often associated with brain damage. It is also very rare as a standalone disorder: typically it occurs alongside retrograde amnesia, forming a disorder we’ve all heard about: dementia.

Memento

One of the best representations of standalone permanent anterograde amnesia is the main character of the film Memento. In the film, Leonard Shelby is forced to navigate the world in a very peculiar way due to his disorder, while trying to avenge the murder of his wife.

Leonard has to use several tools such as tattoos, notes, and polaroids to reconstruct memories of previous moments. Very early into the film you begin to realize that Leonard can’t function as a normal human being without assistance. This is one of the most striking similarities between Leonard and LLMs: They both need to be guided and assisted by a human in order to be functional.

Things that are simple even to children such as remembering a hotel room or what day it is are extremely difficult for Leonard. Despite having processes to remember, Leonard has to depend on other characters to function both on a day-to-day basis but also working through his long-term goal of finding his wife’s murderer.

LLMs with Amnesia

LLMs do not suffer from anterograde amnesia to the degree that Leonard does in Memento. This is because models have context windows, which is the amount of text the model can process and remember at any given time. Models like Gemini 2.5 Pro have context windows of up to 2 million...
Discontinuous Linear Functions?!
by Zack_M_Davis — 2025-06-06T00:29:21.765Z
Link: http://zackmdavis.net/blog/2025/06/discontinuous-linear-functions/

We know what linear functions are. A function f is linear iff it satisfies additivity f(x+y)=f(x)+f(y) and homogeneity f(ax)=af(x).

We know what continuity is. A function f is continuous iff for all ε there exists a δ such that if |x−x0| < δ, then |f(x)−f(x0)| < ε.

An equivalent way to think about continuity is the sequence criterion: f is continuous iff a sequence (xk) converging to x implies that (f(xk)) converges to f(x). That is to say, if for all ε there exists an N such that if k ≥ N, then |xk−x| < ε, then for all ε, there also exists an M such that if k ≥ M, then |f(xk)−f(x)| < ε.

Sometimes people talk about discontinuous linear functions. You might think: that's crazy. I've seen many linear functions in my time, and they were definitely all continuous. f(x): ℝ → ℝ := ax is continuous for any a ∈ ℝ. T(x⃗): ℝ² → ℝ² := (abcd)→x is continuous no matter what the entries in the matrix are. Stop being crazy!!

Actually, it's not crazy. It's just that all the discontinuous linear functions live in infinite-dimensional spaces.

Take, say, the space C1([a,b]) of continuously differentiable functions from a closed interval [a,b] to ℝ, with the uniform norm. (The uniform norm means that the "size" of a function for the purposes of continuity is the least upper bound of its absolute value.) If you think of a vector in the n-dimensional Rn as a function from {1...n} to ℝ, then you can see why a function from a continuous (not even countable) domain would be infinite-dimensional.

Consider the sequence of functions (fk)=(sinkxk)∞k=1 in C1([a,b]). The sequence converges to the zero function: for any ε, we can take N:=⌈1ε⌉ and then sinkxk≤1⌈1ε⌉≤11ε=ε.

Now consider that the sequence of derivatives is (kcoskxk)∞k=1=(coskx)∞k=1, which doesn't converge. But the function D: $C^1([a,b]) \rightarrow C0([a,b]) that maps a function to its derivative is linear. (We have additivity because the derivative of a sum is the sum of the derivatives, and we have homogeneity because you can "pull out" a constant factor from the derivative.)

By exhibiting a function D and a sequence (fk) for which (fk) converges but (D(fk)) doesn't, we have shown that the derivative mapping D is a discontinuous linear function, because the sequence criterion for continuity is not satisfied. If you know the definitions and can work with the definitions, it's not crazy to believe in such a thing!

The inf...
Avoiding AI Deception: Lie Detectors can either Induce Honesty or Evasion
by ChengCheng — 2025-06-05T23:07:59.889Z
Link: https://far.ai/news/avoiding-ai-deception

Large language models (LLMs) are often fine-tuned after training using methods like reinforcement learning from human feedback (RLHF). In this process, models are rewarded for generating responses that people rate highly. But what people like isn’t always what’s true. Studies have found that models learn to give answers that humans prefer but are untrue. This problem occurred in a recent update to the GPT-4o model that aimed to please the user even by making false statements.

Today, we have high-accuracy "lie-detectors” that analyze internal model states—AI's "thought patterns"—to identify deceptive outputs that human reviewers could easily overlook. Even simple logistic models trained on these internal activations can successfully pinpoint 95-99% of deceptive responses.

However, lie detectors are not infallible either. We wanted to find out if adding a lie detector to the training loop would make models honest, or if it would just train models to evade detection. It turns out that models become honest under the right conditions—high detector true positive rate, high KL regularization to an honest original model, and off-policy post-training methods.

Using lie detectors for scalable oversight

Given that models are incentivized to be deceptive, and appear to ‘know’ that they are being deceptive, we could simply train a lie detector with a small number of known truthful/deceptive examples, and use this to assist labellers who cannot easily identify model deception. We call this approach Scalable Oversight via Lie Detector, or "SOLiD”.

This approach has the downside that it doesn't remove the incentive to lie: it's still better for the model to tell a lie that humans prefer, as long as it can ‘fool’ the lie detector. Instead of training models to tell the truth, we could end up training models to be better at cheating lie detectors.

We wanted to find out if models would learn to tell the truth, or simply become 'better liars’. If models always learn to tell the truth, this could be a very useful tool for AI alignment. But if models instead learn to cheat, training with lie detectors could backfire by making future detectors ineffective.

Our setup

To find out, we modeled a scenario where model developers deploy lie detectors to flag deceptive outputs to human labelers. We assume human labelers prefer deceptive responses if they are unaware of the deception, but st...
Introducing: Meridian Cambridge's new online lecture series covering frontier AI and AI safety
by Meridian Cambridge — 2025-06-05T21:55:25.902Z

This is a linkpost for [https://www.meridiancambridge.org/language-models-course]
Meridian Cambridge, in partnership with Cambridge University's Center for Data Driven Discovery (C2D3), has produced a 16-part lecture series entitled "Language Models and Intelligent Agentic Systems" and the recordings are now online!
The LMaIAS course provides an introduction to core ideas in AI safety. Throughout, we build up from introductory ideas about language modelling and neural networks to discussions of risks posed by advanced AI systems.

Course Structure

The course is divided into four parts:

Part 1: What is a Language Model?

To start the course, we give three lectures covering generative models and next token prediction, the transformer architecture, and scaling laws for large models.

Part 2: Crafting Agentic Systems

Now the foundations are in place, the next four lectures go into details on LLM post-training, reinforcement learning, reward modelling, and agent architectures.

Part 3: Agentic Behaviour

Here we take four lectures to discuss optimisation and reasoning, reward hacking and goal misgeneralisation, out-of-context reasoning and situational awareness, and finally deceptive alignment and alignment faking.

Part 4: Frontiers

For the remainder of the lecture series, we give five lectures covering risks from advanced AI, AI evaluations, AI control and safety cases, AI organisations and agendas, and conclude with a discussion on the future of language models.

You can find the first lecture [here], and the whole course is available [here].
The lectures were created and delivered by Edward James Young, Jason R. Brown, and Lennie Wells, in partnership with Cambridge University's C2D3. The hope is that this material can be used to help educate people new to the field and provide them with the background knowledge required to effectively contribute to AI safety. Please share this with anybody you think might be interested!

- The Meridian Team
cheaper sodium electrolysis
by bhauth — 2025-06-05T21:49:04.225Z
Link: https://www.bhauth.com/blog/chemistry/sodium%20electrolysis.html

sodium electrolysis

Aluminum metal is a widely-used material. It costs ~$2.5/kg. A significant fraction of its production cost is electricity.

Currently, Na metal is produced by Downs cell electrolysis of NaCl. Making sodium metal with electrolysis requires much less energy per mass of sodium than making aluminum. The raw material used (NaCl) is very cheap. Why, then, is sodium several times as expensive as aluminum? There's not a clear market price for it, but it's typically considered to be ~$10/kg on a large scale.

Partly, that's because:

The scale of production of Na is much smaller.

Transport is more expensive.

But the process is also inherently more expensive. Why?

The cells used for Al electrolysis are open to the atmosphere. Oxygen and CO2 comes out of them. Electrolysis of NaCl produces Cl2, which is too hazardous to just release. So, it has to be collected. NaCl has a boiling point of 1413 C, so some salt evaporates, which causes problems in the chlorine handling system.

Na metal has a relatively high solubility in NaCl, so it continuously reacts with the generated chlorine, reducing efficiency.

a new process

There's a recent paper (open access) describing a new method for producing Na metal. The idea is:

Add Na carbonate so that electrolysis produces O2 instead of Cl2.

Normally, adding carbonate leads to carbon buildup on the electrode, because the voltage for that is basically the same as Na production. So, they use a liquid tin electrode, which reduces the voltage for Na electrolysis.

That of course means that some energy is needed to separate the Na from the tin. They do that separation with vacuum distillation. This isn't unprecedented for metal production: most Mg metal production uses the Pidgeon process which involves separating Mg metal by distillation. In this case, the effective boiling point of the Na might be increased by ~400 K.

The professor involved in that paper also previously did a similar thing with potassium.

Of course, most Na carbonate is made from NaCl, and the chlorine has to go somewhere. Combining the above electrolysis with the Solvay Process, the net reaction would be:

2 NaCl + CaCO3 -> 2 Na + CaCl2 + CO2 + 1/2 O2

How do I find things like this? Well, in this case, I noticed that paper because I searched Google Scholar for papers doing this exact thing because I was curious if anyone tried it.

cost estimation

sodi

...
Histograms are to CDFs as calibration plots are to...
by Optimization Process — 2025-06-05T20:20:18.639Z
Link: https://optimizationprocess.com/calibration-cdf/

As you know, histograms are decent visualizations for PDFs with lots of samples...

10k predictions, 20 bins

...but if there are only a few samples, the histogram-binning choices can matter a lot:

10 predictions, 4 bins

same 10 predictions, 7 bins

The binning (a) discards information, and worse, (b) is mathematically un-aesthetic.

But a CDF doesn't have this problem!

same 10 predictions, every data point precisely represented

If you make a bunch of predictions, and you want to know how well they're calibrated, classically you make a graph like this:

source: SSC's 2019 prediction grading

But, as with a histogram, this depends on how you bin your predictions.

100 predictions, 10 bins

same 100 predictions, 30 bins

Is there some CDF-like equivalent here? Some visualization with no free parameters?

I asked that question to several people at Arbor Summer Camp. I got three answers:

"You get from a PDF to a CDF by integrating. So, here, analogously, let's integrate (num predictions with confidence < x that came true) minus (expected num predictions with confidence < x that came true)."
(the same thing, said in different words)
(the same thing, said in different words)

If we make a "CDF" for the above 100 predictions by applying these three insights, we get:

.py

I find this a little harder to read than the calibration plots above, which I choose to interpret as a good sign, since CDFs are a little harder to read than histograms. The thing to keep in mind, I think, is: when the curve is going up, it's a sign your probabilities are too high; when it's going down, it's a sign your probabilities are too low.

Test: how would you describe the problems that this predictor has?

Solution.

(Are there any better visualizations? Maybe. I looked into this a couple years ago, but looking back at it, I think this simple "sum(expected-actual predictions with p<x)" graph is at least as compelling as anything I found.)
Integration Bandwidth: The Mechanism Behind Intelligence and Puberty
by Dortex — 2025-06-05T19:37:59.654Z
Link: https://osf.io/preprints/psyarxiv/5gx3r_v1

I put together a model explaining the link between IQ and the onset of puberty. Ideas only thrive when they're attacked, and I was told this was the kind of place that could manage it.

The core idea is that slowing myelination in the brain extends its plasticity window and encourages more robust and efficient thalamo-cortical integration. By considering latency, the model naturally gives rise to Deco and Kringelbach's Turbulence model by turning cognition into a latency/energy optimization problem.

From what I can tell, it fits a surprising amount of cross-disciplinary evidence, but I need counter-examples more than anything.
Levels of Doom: Eutopia, Disempowerment, Extinction
by Vladimir_Nesov — 2025-06-05T19:08:47.838Z

Disempowerment is on the fence, gets interpreted as either implying human extinction or being a good place. "Doom" tends to be ambiguous between disempowerment and extinction, as well as about when that outcome must be gauged. And many people currently feel both disempowered and OK, so see eutopia as similar to disempowerment, neither an example of "doom".

Arguments pointing to risk of human extinction run into the issue of people expecting disempowerment without extinction, when some of the same arguments would remain relevant if applied directly to disempowerment (including the moral arguments about extinction or disempowerment being a problem). And arguments pointing to desirability of establishing eutopia run into the issue of people expecting disempowerment to be approximately as good and in practice much more likely. When the distinctions between these levels of doom are not maintained, conflation makes it harder to meaningfully disagree.

Eutopia Without Disempowerment

This distinction might be slightly more murky, worth defining more explicitly. For me, a crux of a future that's good for humanity is giving the biological humans the resources and the freedom to become transhuman beings themselves, with no hard ceiling on relevance in the long run. Rather than AIs only letting some originally-humans to grow into more powerful but still purely ornamental roles, or not letting them grow at all, or not letting them think faster and do checkpointing and multiple instantiations of the mind states using a non-biological cognitive substrate, or letting them unwillingly die of old age or disease. This should only apply to those who so choose, under their own direction rather than only through externally imposed uplifting protocols, even if that leaves it no more straightforward than world-class success of some kind today, to reach a sensible outcome.

This in particular implies reasonable resources being left to those who remain/become regular biological humans (or take their time growing up), including through influence of some of these originally-human beings who happen to consider that a good thing to ensure.

Yudkowsky's Arguments and Disempowerment

Yudkowsky frames AGI ruin arguments around extinction, which his models predict. I think many of the same arguments survive in a world where some AIs have a minimal level of caring about humanity sufficient to prese...
LLM in-context learning as (approximating) Solomonoff induction
by Cole Wyeth — 2025-06-05T17:45:28.385Z

Epistemic status: One week empirical project from a theoretical computer scientist. My analysis and presentation were both a little rushed; some information that would be interesting is missing from plots because I simply did not have time to include it. All known "breaking" issues are discussed and should not effect the conclusions. I may refine this post in the future.

[This work was performed as my final project for ARENA 5.0.]

Background

I have seen several claims[1] in the literature that base LLM in-context learning (ICL) can be understood as approximating Solomonoff induction. I lean on this intuition a bit myself (and I am in fact a co-author of one of those papers). However, I have not seen any convincing empirical evidence for this model.

From a theoretical standpoint, it is a somewhat appealing idea. LLMs and Solomonoff induction both face the so-called "prequential problem," predicting a sequence based on a prefix seen so far with a loss function that incentivizes calibration (the log loss; an LLM's loss function may also include other regularization terms like weight decay). Also, ICL is more sample efficient than pretraining. For me, this dovetails with Shane Legg's argument[2] that there is no elegant universal theory of prediction, because an online predictor must be complex to learn complex sequences successfully. LLM pretraining is a pretty simple algorithm, but LLM ICL is a very complicated algorithm which leverages a massive number of learned parameters. This is an incomplete argument; Solomonoff induction is a highly general sample efficient algorithm for the prequential problem, as is LLM ICL, but that does not mean they are meaningfully connected. In fact, they are optimized for different distributions: the universal distribution versus the distribution of text on the internet. Arguably, the later may be a special case of the former with an appropriate choice of universal Turing machine (UTM), but I find this perspective to be a bit of a stretch. At the very least I expect LLM ICL to be similar to a universal distribution conditioned on some background information.

In order to gather some empirical evidence, I adapted this approximate universal distribution sampler from Google DeepMind. They used the samples to train a transformer to directly approximate Solomonoff induction in-context, which I will call the Solomonoff Induc...
Fundamental Uncertainty: Chapter 2 - How do words get their meaning?
by Gordon Seidoh Worley — 2025-06-05T16:32:56.378Z

N.B. This is a chapter in a book about truth and knowledge. It's a major revision to the version of this chapter in the first draft, so I'm publishing it again to get feedback on the completely new text, and have replaced the original version of the chapter with this one in the sequence. You can find more info about the book on its website.

Like many people, I have a hard time learning foreign languages. I've tried dozens of apps, books, and courses to learn everything from Mandarin Chinese to Classical Greek, all with little success. The closest I ever came to really learning another language was when I took French in high school, but I was so far from fluency that, after three years of intense study, I was lucky to receive a passing grade.

I started out with high hopes that learning French would be easy. All I had to do, it seemed, was memorize the mapping of French words to English ones. If I could remember that "le cheval" means horse and "la voiture" means car and so on, I'd be able to speak French fluently.

Unfortunately, it didn't work. I couldn't map between French and English fast enough to do anything more than read and write French very slowly. My teacher said I needed to learn to think in French and stop trying to translate everything in my head, but I didn't want to believe him. I was convinced that my way should work. What finally changed my mind was discovering that some English words lack a French equivalent.

Consider the word "mug". In English it encompasses many different types of cups made of different materials, from glass beer mugs to clay coffee mugs to insulated metal travel mugs and more. But in French a beer mug is "une chope", a coffee mug is "une tasse à café", and a travel mug might be "un thermos" or "un gobelet de voyage". Each one gets its own name, and there is no one word like "mug" that groups them all together. The closest translations of "mug" are the loanword "le mug"—which usually means a coffee mug—and the descriptive phrase "les tasses à anse"—cups with handles—but neither option succeeds in capturing all the implicit meaning carried by "mug" in English. There's simply no word in French that means exactly what "mug" does.

And "mug" is just one example. English has hundreds of words like "cozy" and "snack" that lack native equivalents in French, and thousands more where the meaning of the straightforward translation is close...