trentmkelly/LessWrong-43k huggingface dataset:
A short project on Mamba: grokking & interpretability
by Alejandro Tlaie — 2024-10-18T16:59:45.314Z

Epistemic status: I've worked on this project for ~20h, on my free time and using only a Colab notebook.

Executive summary

I trained a minimalistic implementation of Mamba (details below) on the modular addition task. I found that:

This non-transformer-based model can also exhibit grokking (i.e., the model learns to generalise after overfitting to the training data).

There are tools that we can import from neuroscience that can help us interpret how the network representation changes as grokking takes place over training epochs.

Introduction

Almost all of the Mechanistic Interpretability (MI) efforts I've seen people excited about and the great majority of the techniques I've learned are related to Transformer-based architectures. At the same time, a competitive alternative (Mamba) was recently introduced and later scaled. To me, when coupling these two facts together, a giant gap between capabilities and safety emerges.

Thus, I think Mamba provides an interesting use case where we can test whether the more conceptual foundations of MI are solid (i.e., somewhat model-agnostic) and, therefore, whether MI can potentially survive another transformer-like paradigm shift on the race towards AGI.

For a bit more of context, Mamba is based on a special version of State Space Models (SSMs): add another S (for Structured) and you have one of its essential components. The actual architecture is slightly more complex, as you can see in this awesome post, than the S-SSM layer, but for this project I wrote up a minimal implementation that could get the job done.

A simple-yet-interesting enough task

The task that the model has to solve is: given two input integers (x and y), return whether their sum is divisible by a big prime number (p=113, in this case). This is mapped into a setup that Autoregressive token predictors can deal with: one input example would consist of three tokens: 'x', 'y' and '=', and the only output token would be either '0' (if (x+y) mod p ≠0) or '1' (otherwise).

My main reason for choosing this oddly specific task was that it's probably the most well understood and reversed engineered example using MI and simple transformers. In other words, I didn't have to re-invent the wheel.

A minimalistic implementation of the SSM layer

My MinimalMambaSSM implementation is a simplified version of the Mamba architecture, combining state-space modelling with gated ...
LLMs can learn about themselves by introspection
by Felix J Binder — 2024-10-18T16:12:51.231Z

Are LLMs capable of introspection, i.e. special access to their own inner states?
Can they use this access to report facts about themselves that are not in the training data?
Yes — in simple tasks at least!

TLDR: We find that LLMs are capable of introspection on simple tasks. We discuss potential implications of introspection for interpretability and the moral status of AIs.

Paper Authors: Felix Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, Owain Evans

This post contains edited extracts from the full paper.

Abstract

Humans acquire knowledge by observing the external world, but also by introspection. Introspection gives a person privileged access to their current state of mind (e.g., thoughts and feelings) that is not accessible to external observers. Can LLMs introspect? We define introspection as acquiring knowledge that is not contained in or derived from training data but instead originates from internal states. Such a capability could enhance model interpretability. Instead of painstakingly analyzing a model's internal workings, we could simply ask the model about its beliefs, world models, and goals.

More speculatively, an introspective model might self-report on whether it possesses certain internal states—such as subjective feelings or desires—and this could inform us about the moral status of these states. Importantly, such self-reports would not be entirely dictated by the model's training data.

We study introspection by finetuning LLMs to predict properties of their own behavior in hypothetical scenarios. For example, "Given the input P, would your output favor the short- or long-term option?" If a model M1 can introspect, it should outperform a different model M2 in predicting M1's behavior—even if M2 is trained on M1's ground-truth behavior. The idea is that M1 has privileged access to its own behavioral tendencies, and this enables it to predict itself better than M2 (even if M2 is generally stronger).

In experiments with GPT-4, GPT-4o, and Llama-3 models (each finetuned to predict itself), we find that the model M1 outperforms M2 in predicting itself, providing evidence for introspection. Notably, M1 continues to predict its behavior accurately even after we intentionally modify its ground-truth behavior. However, while we successfully elicit in...
Low Probability Estimation in Language Models
by Gabriel Wu — 2024-10-18T15:50:05.947Z

ARC recently released our first empirical paper: Estimating the Probabilities of Rare Language Model Outputs. In this work, we construct a simple setting for low probability estimation — single-token argmax sampling in transformers — and use it to compare the performance of various estimation methods. ARC views low probability estimation as a potential technique for mitigating worst-case behavior from AI, including deceptive alignment; see our previous theory post on estimating tail risks in neural networks for further discussion.

Outline of this post:

Problem statement and description of our setting

Estimation methods we compared: importance sampling and activation extrapolation

Summary of empirical results

Discussion, including motivation, limitations, related work, and future directions

This document describes joint research done with Jacob Hilton, with contributions from Paul Christiano (while at ARC prior to April 2024). There were also intermediate theoretical contributions from David Matolsci and George Robinson. Thanks additionally to Jean-Stanislas Denain and Eric Neyman for feedback.

Problem statement

Given a model M:X→Y, an input distribution D∈Δ(X), and a computable boolean property of outputs C:Y→{0,1}, low probability estimation is the task of efficiently estimating

Prx∼D[C(M(x))=1],

even when this probability is too small to detect via naive sampling. For example, in ARC's previous post on estimating tail risks, C represented a "catastrophe detector" that can tell us if an action M(x) proposed by the model would result in large-scale harm.

In this work, we study low probability estimation in the setting of argmax sampling from language models with single-token behaviors. Let M:V∗→V be a transformer language model that predicts the next token given a string of previous tokens, where V is the token vocabulary. Note that we sample at temperature 0, so M is deterministic. Given a distribution D over V∗ and a target token t∈V, the low probability estimation problem for single-token behaviors is the task of estimating

Prx∼D[M(x)=t]

In general, D can be any distribution that can be formally specified, such as the output distribution of generative model. However, in this paper we focus only on distributions D in which every token position is independent.[1] This is a pretty restrictive choice, and we expect future work to move beyond this assumption. H...
The Mysterious Trump Buyers on Polymarket
by Annapurna — 2024-10-18T13:26:25.565Z
Link: https://jorgevelez.substack.com/p/the-mysterious-trump-buyers-on-polymarket

How four accounts have shifted the election probabilities narrative

As the US Presidential Election approaches, all eyes are on who will become the next president - Vice President Kamala Harris or former President Donald Trump. In early October, many analysts believed both candidates had a relatively equal chance of winning. However, a significant shift in betting markets occurred following Trump’s rally in Butler, PA on October 6, with odds heavily moving in his favor.

On

Polymarket

, traders began noticing a sharp increase in bets on Trump. This shift was driven by a group of relatively new accounts, which have, as of today, deposited approximately $35 million USDC into Polymarket and deployed $33 million across election-related markets, always betting on Trump / Republican outcomes. This has sparked a lot of online speculation, leading to misconceptions. Below, I aim to clarify these points.

Source: Arkham Intelligence

1. Can a Group of Accounts Really Shift the Market This Much?

While Polymarket’s US Presidential Election market might be the most liquid in terms of political betting today, it isn’t as deep as it appears. For example, the bid-ask spread sits at 58,657 x 60.6 - 60.7 x 35,429. This represents a relatively tight market, but one without significant liquidity. If someone were to place a market order for 1 million Trump shares right now, the odds would move from 60.7 to 62. Since October 6, these accounts have collectively bought over 40 million Trump shares, leading to the observed shift in Trump’s odds from 50.8 to 60.7.

Source: Polymarket

2. Is Trump Simply Gaining Momentum, and Did These Accounts Predict That?

Possibly. Many market participants are investigating whether these accounts are related, and if so, who might be behind them. Understanding who might be behind these accounts can provide clues as to whether they have superior knowledge and resources when it comes to the election. Some users on X have criticized attempts to identify the individuals, arguing that it is unethical or illegal. However, understanding who your counterparty is can provide a competitive advantage, a tactic frequently employed in capital markets. For example, when a large block of shares enters the market, traders often ask, "Who is selling?" to inform their strategies.

3. Why Aren’t Sharp Traders Betting the Other Side to Normalize the Odds?

Some market participants have...
On Intentionality, or: Towards a More Inclusive Concept of Lying
by Cornelius Dybdahl — 2024-10-18T10:37:32.201Z

Intentional actions may be contrasted with unintentional actions, which are generally the result of either instinct or accident. More precisely, we may say that unintentional actions are either the result of some unconscious cognitive process that culminates in that action (eg. habits, instincts), or they are not, as is the case with accidents and primitive reflexes. An intentional action, on the other hand, is the result of a cognitive process, but one that aims at a definite goal.

To take an example from the LessWrong canon, suppose you are purchasing a trunk in which to keep your school supplies. You like the shiny one, but cannot quite permit yourself to choose the trunk on this basis, so you come up with all sorts of reasons about why that trunk is the best one — all these reasons being unrelated to its shine, of course. But ultimately you still picked it because it was shiny, for all that you have rationalised your way into not even noticing that that was your real motive for choosing that particular trunk.

The fact that an action is driven by a particular intent does not imply that you must be reflectively aware of that intent. Actually, the very fact that it is possible to gain awareness of your intents by introspection seems to imply that the intents exist prior to becoming aware of them.

Now consider how this relates to lying — we shall take another classic LessWrongian example: that of religious belief-in-belief: a person who thinks he believes in God but actually doesn't, as revealed by his overall behaviour and other aspects of his worldview. In the course of consoling a troubled friend, he might declare "God does exist and he really does answer prayers." If you were to accuse him of lying, he would be quite aghast because he can readily point to lots of evidence that he is indeed a Christian: he goes to church every Sunday, he prays every evening before supper, he knows a lot of church liturgy, listens to Christian rock, and so on.

But the fact remains that he intentionally made an assertion that he does not believe to be true. I must choose my words carefully here: it is not that he intended to make a false statement, which would imply he has reflected on the statement itself and deemed it untrue. It is not the statement but its content that he disbelieves in. He simply has not quite realised that the statement relates to such content; that ...
Species as Canonical Referents of Super-Organisms
by Yudhister Kumar — 2024-10-18T07:49:52.944Z
Link: https://www.yudhister.me/species-as-canonical-referents-of-super-organisms/

A species is a reproductively isolated population. In essence, it consists of organisms which can only breed with each other, so its ability to self-replicate is entirely self-contained. In practice, the abstraction only applies well to macroflora and macrofauna, which is still enough to inform our intuitions of super-organismal interaction.

Interspecific interactions can frequently be modeled by considering the relevant species as agents in their own right. Agents motivated by self-sustention to acquire resources, preserve the health of their subagents, and bargain or compete with others on the same playing field as themselves. Parasitism, predation, pollination—all organismal interactions generalizable to super-organismal interactions.

Optimization of the genome does not occur at the level of the organism, nor does it occur at the level of the tribe. It occurs on the level of the genome, and selects for genes which encode traits which are more fit. From this perspective, it makes sense for "species" to be a natural abstraction. Yet, I claim there are properties which species have that make them particularly nice examples of super-organisms in action. Namely:

Boundaries between species are clear and well-defined, due to reproductive isolation;
Competitive dynamics between species are natural to consider, rather than having to move up or down a vertical hierarchy;
The "intentional stance", when applied to species, is simple: reproduction.

However, it is precisely because species have such nice properties that we should be incredibly cautious when using them as intuition pumps for other kinds of super-organisms, such as nation-states, companies, or egregores. For instance:

Boundaries between nation-states and companies are relatively straightforward to define (determined by citizenship or residency and employment, respectively). Boundaries between egregores are . . . complicated, to say the least.[1]
Company competition is generally modelable with agent-agent dynamics, and so is nation-state competition. But the act of "merging" (via acquisition, immigration, etc.) is available to them in a way that it is not to species. (Again, egregores are complicated . . .)
The goal of a company is to maximize shareholder value. The goal of a nation-state is . . . to provide value to its citizens? The "goal" of an egregore is ostensibly to self-perpetuate and . . . fullfill whiche

...
NAO Updates, Fall 2024
by jefftk — 2024-10-18T00:00:04.142Z
Link: https://naobservatory.org/blog/updates-fall-2024

It's been a busy season at the Nucleic Acid Observatory, and we have a lot to share since our last update. As always, If anything here is particularly interesting or if you’re working on similar problems, please reach out!

Wastewater Sequencing

We performed an initial analysis of untargeted sequencing data from aggregated airplane lavatory waste and municipal treatment plant influent that we collected and processed during our Fall 2023 partnership with CDC’s Traveler-based Genomic Surveillance program and Ginkgo Biosecurity. We've now analyzed viral abundance and diversity in sequencing data across multiple sample types and wastewater-processing and sequencing protocols. Next steps include further investigating how protocol and sample-type affect specific viruses and bacteria, as well as understanding pathogen temporal dynamics seen in airport versus treatment-plant samples.

We have continued to work with Pardis Sabeti's Lab at the Broad Institute to develop an optimized protocol for preparing RNA sequencing libraries from nucleic acids extracted from wastewater. By the end of the year, we plan to use this protocol to sequence the samples collected during our Fall 2023 effort described above.

We have also been collaborating with Jason Rothman, formerly of Katrine Whiteson's lab at the University of California, Irvine, and now at his own lab at the University of California, Riverside, to sequence and analyze Southern California wastewater. The sequencing is complete, with a total of 45B read pairs. We're aiming to make the raw sequencing data public with a data paper later this fall.

We've scaled up our collaboration with Marc Johnson's group at the University of Missouri. Marc's group is now running a full flow cell every other week. While the MU Genomics Core had been sequencing these on a NovaSeq 6000 with the S4 flow cell they've moved to the NovaSeq X+ with the 10B flow cell, which lowers sequencing costs by about 30% due to cheaper reagents. These runs include samples from sewersheds in multiple major metropolitan areas, among them four Chicago-area sewersheds through a new collaboration with Rachel Poretsky at University of Illinois Chicago. We're still looking for additional partnerships, and if you have a relationship with a treatment plant in a city with a large amount of international travel we can potentially sequence your influent and share the data w...
You're Playing a Rough Game
by jefftk — 2024-10-17T19:20:06.251Z

In general, we don't want our kids (10y, 8y) to hit each other.
Learning to control your impulses is an important skill, and resorting
to violence is usually a substitute for other skills we're prefer them
to practice (understanding what the other person wants, negotiating).
Also they could hurt each other.

On the other hand, sometimes the kids enjoy hitting each other. This
is very different from hitting out of anger: they're both having fun,
they're not trying to injure each other, it's more about force than
impact, etc. Even calling both of these activities "hitting" is a bit
misleading: a hit intended to inflict pain looks very different than
one intended to knock the other off balance or push them farther away
on the couch to gain a strategic advantage.

We wouldn't want to prohibit our kids from playing roughly with each
other when that's what they both want, but this interacts awkwardly
with normal rules. If Lily says "Anna hit me" but this was after Lily
said "Anna, lets play a game where we hit each other" then Anna should
clearly not go in time
out. The way we generally handle this is flagging to the kids when
we notice they're doing this ("it looks like you're playing a rough
game") and possibly including a warning ("and someone might get
hurt"). Then if someone does get hurt, and minor injuries are
reasonably common with this sort of play, they know we're not going to
punish the other person for it ("you were playing a rough game, and
this is the kind of thing that can happen when you're playing rough").
If they were doing this upstairs or somewhere we didn't notice we do
our best to figure out what happened ("What were you doing before they
hit you? ... It sounds like you were playing a rough game.") but it's not
perfect.

One way this could be abused is to use the context of a rough game to
escalate to actually trying to hurt the other person. How this works
out sounds like something that would vary a lot based on the actual
kids involved, but with ours this is rare: I don't see them using
rough games as cover for malice. Much more common are issues with not
being on the same page about whether they're playing a rough game. We
sort that out as best as we can ("Anna, is this a game you want to be
playing?") and try to encourage them to do this on their own ("Does it
look like Lily's enjoying this?").

I'm also curious how this...
Factoring P(doom) into a bayesian network
by Joseph Gardi — 2024-10-17T17:55:37.455Z

I wouldn't be surprised if this is already a thing so please let me know if it is. I have tried searching. I'd like a tool like this one for analyzing P(doom): https://projects.fivethirtyeight.com/2024-election-forecast/. Other precedents are this model of transformative AI timeline on less wrong and the use of bayesian networks for forecasting climate change.

The problem of estimating P(doom) is very complicated but it is not impossible. It is not metaphysical like estimating P(we live in a simulation) or P(we are a boltzman brain). P(doom) is a tangible thing based on human technology. Just very very complicated. It requires summing over many different possibilities. So maybe can we do better by factoring the distribution? This would break the problem down into parts which can each be analyzed one at a time.

Suppose there were market or expert based priors for the following probabilities.

P(a single entity would lose control of the AI). This one component has been the main focus of safety research.
P(the international community can prevent the development of ASI)
P(the international community can restrict number of entities that have ASI)
P(doom | ASI is easily available to all)
P(number of entities that have ASI)
P(a single rogue AI could destroy humanity despite good AI's trying to defend humanity)

P(a single human group would use ASI to oppress the rest of the world)

One could build a bayesian network from those priors and estimate P(doom) using one of many methods (e.g. variational inference).

The list would have to go on much longer actually. And the model would be more complicated.

There are blog posts and papers analyzing each of the probabilities already. It is just a matter of putting all this accumulated theory together into a single model. On the On the other hand, one subtle fallacy could result in absurd results. There could be error compounding. The analysis could fail to include all possibilities. It could be very sensitive to hyper-parameters.

Now some would argue that these challenges are why it's so important for an argument to be simple. But I think these challenges just mean we need many people thinking very carefully about every detail of the model.