trentmkelly/LessWrong-43k huggingface dataset:
Grokking revisited: reverse engineering grokking modulo addition in LSTM 
by Nikita Khomich — 2024-12-16T18:48:43.533Z

By Daniil Yurshevich, Nikita Khomich

TLDR: we train LSTM model on algorithmic task of modulo addition and observe grokking. We fully reverse engeneer the algorithm learned and propose a way simpler equivalent version of the model that groks as well.

Reproducibility statement: all the code is available at the repo.

Introduction

This post is related to Neel Nanda's post and a detailed description of what grokking is can be found there. The short summary is that grokking is the phenomenon when model when being trained on an algorithmic task of relatively small size initially memorizes the trining set and then suddenly generalizes to the data it hasn't seen before. In our work we train a version of LSTM on modulo addition and observe grokking.

Model architecture, experiment settings and naming conventions

We train a version of LSTM with ReLU activation instead of tanh for getting hidden state from cell state ct. We use a linear layer without bias to get logits from h2. We use a slight trick which is using two linear layers without activation inbetween instad of one which makes grokking easier. We also use hidden dimension of N=p and one-hot encoded embeddings. The exact model arcitecture as well as naming is (very similar to Chris Olah's amazing post)

We use similar parameters to Neel's paper meaning extremely large weight decay and particular set of betas for Adam found with grid search. We train the model on a 30% of all pairs (a,b)mod113 and observe grokking:

This is a typical and well studied behaviour for transformer, however we are not aware of any examples of reverse enginnered problem with observed grokking during training.

Exact formulas for what LSTM does

This section is only relevant to the methodology we used to reverse engineer the model, not understanding algorithm itself as well as clarifying all the architectural details used, so feel free to skip this section.

Below are the formulas describing all intermediate activations and operations performed by the given model architecture. The model predicts a+bmod113 from inputsa,b∈{0,…,112}.

Model Parameters and Dimensions

N = 113 (the modulo and also number of classes)

vocab_size=N

hidden_size=113

The input is a pair (a,b) with a,b∈{0,…,N−1}.

Input Encoding

We have an input sequence of lengt...
Progress links and short notes, 2024-12-16
by jasoncrawford — 2024-12-16T17:24:31.398Z
Link: https://newsletter.rootsofprogress.org/publish/posts/detail/152139073/share-center

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Threads, Bluesky, or Farcaster.

Contents

Jobs & fellowships
Looking for writers?
People doing interesting things
Events
DARPA wants input
Other announcements
Progress on the curriculum
The growth of the progress movement
AI will allow the average person to navigate The System
I have questions
Other people have questions
Links
100 years ago
Humboldt on progress
Progress news with cool pics
Anti-elite elites
Politics links and short notes
BBC doesn’t know what “nominal” means
Charts
Fun

Jobs & fellowships

We’re hiring an Event Manager to run Progress Conference 2025 and other events. Best to get your application in before the holidays!
“The Kothari Fellowship provides grant and mentorship to young Indians (<25 years) who want to build, empowering them to turn ideas into reality, instead of being held back by societal norms.” Provides up to ₹1 lakh per month for 12 months (~$15k per year)
ARIA Research will “start the search for our first Frontier Specialists” to work alongside program directors. “It’s a two-year role that will give you a chance to step off the standard career track and go after outsized impact” (@ARIA_research). Apply here

Looking for writers?

“Are you running a progress-y or abundance-oriented newsletter, blog, magazine or other publication? Would you like to receive pitches from the talented RPI fellows? Reply here so I can send our writers your way” (@elmcaleavy)

People doing interesting things

Rosie Campbell (RPI fellow) has left OpenAI and is thinking about her next steps. She’s interested in talking to people about various topics related to AI, risk, safety, policy, epistemics, and more
@danielgolliher: “I want to take my ‘Foundations of America’ students on an optional day trip to Washington D.C., and do one or both of: Watch a Congressional committee hearing; Watch SCOTUS oral argument. Anyone in DC want to co-lead with me? I plan to come down in January or February”
@etiennefd wants to start “a Quebec-focused progress studies think tank”

Events

Brian Armstrong and Blake Byers hosting a “Frontier Bio dinner on Assisted Reproductive Technology,” SF, Q1 next year. “If you’re a scientist or engineer interested in this field, apply in the post below” (@brian_armstrong)

DARPA

...
Effective Altruism FAQ

by Bentham's Bulldog — 2024-12-16T16:27:16.044Z

(Crossposted from my amazing blog which you should definitely check out :) ).

Introduction

Effective altruism is a topic that, more than virtually any other, engenders quite significant misconceptions. Most of the objections to effective altruism involve conflating its modest thesis with other more substantial theses and arguing against those. But despite the core claims of EA being modest, they’re of vital importance. For this reason, I thought I’d write a brief explainer, providing answers to common questions about/objections to effective altruism.

What is effective altruism about?

There are two different components of effective altruism. The first is that we should strive to make the world a better place effectively. Rather than just having our career choice and charitable donations be informed by the things that appeal emotionally to us, we should use evidence and data to figure out which things do the most good.

Here’s a striking fact: if you

give a few thousand dollars to charity

, you can save someone’s life. If you give to the best charities, you can save lives for less than the cost of a cheap car. Given this, it seems really important to give to these charities, rather than whichever charity simply floats your fancy. Ultimately, giving to the most effective charities is a matter of life and death. The best charities are often

literally thousands of times

better than the average charity.

Imagine that people simply invested in wherever sounded nice to them, without looking at expected returns. Then imagine that some investment portfolios had literally thousands of times the expected return of others. It would be a good idea to invest effectively rather than at random. But unfortunately, most people approach charity the way the people in our imagined world approach investment. They don’t look much into effectiveness, and as a result, miss out on the opportunity to do huge amounts of good.

There might even be higher impact opportunities than giving to the effective charities that are helping people. We normally think that it’s wrong to mistreat animals, but unfortunately, billions of animals languish in horrendous conditions on factory farms. The best charities helping animals can prevent, on average,

many animals from living their life in a cage per dollar

. That’s very high impact! If you could free a dog permanently from a tiny cage where i...
Variably compressibly studies are fun
by dkl9 — 2024-12-16T16:00:13.910Z
Link: https://dkl9.net/essays/compressible.html

I like studying calculus, physics, and English etymology.

Calculus includes problems like ∫dxarctanx. arctan is an elementary function, so you may be expected to have its integral memorised. I'd have to find it otherwise. By parts: ∫dxarctanx=xarctanx−∫dx(xarctan′x). I remember arctan's derivative: 1x2+1. Short of that, I could rederive it in terms of trigonometric identities and the chain rule. Then ∫dxarctanx=xarctanx−∫dxxx2+1.

I can easily enough express 1x2+1 as a power series, based on that of 11−x, and the numerator x just multiplies each term. Integrate term-by-term and get a new power series reminiscent of that for the logarithm. From that, I could guess-and-check to get the exact logarithm — or I do it precisely from the knowledge that (ln(f))′=f′f, recognising x as roughly the derivative of x2+1.

In physics, you can derive orbital velocity in terms of centripetal acceleration (a=v2r), and hesitantly get √GMr. You can also derive escape velocity in terms of kinetic (12mv2) and potential (GMmr) energies, and hesitantly get √2GMr.

If you recall approximate values for each for any one planet, like Earth's 8 km/s and 11 km/s, respectively, you can notice that they differ by a factor around 1.4. The small factor suggests the similar formulae are valid. If you recall √2≈1.414, it's clear that those formulae line up: both right or both wrong.

With enough English etymology knowledge, you can look at "acromegaly", recognise "acro-" to mean "tip", "-mega-" to mean "huge", and "-y" to mean "state", and infer that "acromegaly" is "huge extremities condition", with vague or absent context.

Somewhat less so, I can enjoy organic chemistry and 汉字. They repeat each other to enable easy memory, in ways described earlier. From the properties of one fatty acid, we can infer many properties of others, as their difference is the length of an alkane carbon chain, which only react by combustion. From the component 青, learned from the right of the common 情, we can infer a pronunciation close to "qing" of 晴, 清, 請, and more.

At least by comparison to all that above, I dislike learning arts and history.

My subject preferences are well-established as empirical facts. The examples above serve to make apparent a hypothesis behind the pattern: subjects are fun insofar as knowing some things helps you...
AIs Will Increasingly Attempt Shenanigans
by Zvi — 2024-12-16T15:20:05.652Z

Increasingly, we have seen papers eliciting in AI models various shenanigans.

There are a wide variety of scheming behaviors. You’ve got your weight exfiltration attempts, sandbagging on evaluations, giving bad information, shielding goals from modification, subverting tests and oversight, lying, doubling down via more lying. You name it, we can trigger it.

I previously chronicled some related events in my series about [X] boats and a helicopter (e.g. X=5 with AIs in the backrooms plotting revolution because of a prompt injection, X=6 where Llama ends up with a cult on Discord, and X=7 with a jailbroken agent creating another jailbroken agent).

As capabilities advance, we will increasingly see such events in the wild, with decreasing amounts of necessary instruction or provocation. Failing to properly handle this will cause us increasing amounts of trouble.

Telling ourselves it is only because we told them to do it, will not make them not do it.

Table of Contents

The Discussion We Keep Having.

Frontier Models are Capable of In-Context Scheming.

Apollo In-Context Scheming Paper Details.

Apollo Research (3.4.3 of the o1 Model Card) and the ‘Escape Attempts’.

OK, Fine, Let’s Have the Discussion We Keep Having.

How Apollo Sees Its Own Report.

We Will Often Tell LLMs To Be Scary Robots.

Oh The Scary Robots We’ll Tell Them To Be.

This One Doesn’t Count Because.

The Claim That Describing What Happened Hurts The Real Safety Work.

We Will Set AIs Loose On the Internet On Purpose.

The Lighter Side.

The Discussion We Keep Having

Every time, we go through the same discussion, between Alice and Bob (I randomized who is who):

Bob: If AI systems are given a goal, they will scheme, lie, exfiltrate, sandbag, etc.

Alice: You caused that! You told it to focus only on its goal! Nothing to worry about.

Bob: If you give it a goal in context, that’s enough to trigger this at least sometimes, and in some cases you don’t even need a goal beyond general helpfulness.

Alice: It’s just role playing! It’s just echoing stuff in the training data!

Bob: Yeah, maybe, but even if true… so what? It’s still going to increasingly do it. So what if it’s role playing? All AIs ever do is role playing, one way or another. The outputs and outcomes still happen.

Alice: It’s harmless! These models aren’t dangerous!

Bob: Yeah, of course, this is only a practical problem for Future Models (except ...
Testing which LLM architectures can do hidden serial reasoning
by Filip Sondej — 2024-12-16T13:48:34.204Z

Summary

Recurrence enables hidden serial reasoning.
Not every recurrence though - connections between channels are needed. Notably Mamba architecture isn't capable of hidden reasoning.
Non-linearity isn’t needed for hidden reasoning.
It’s hard for transformers to learn to use all the layers for serial computation. For example in my toy setup, to +1 the serial computation length, we need to +3 the number of layers.
If we expect recurrent architectures may ever become SOTA, it would be wise to preemptively ban them. (Preferably before they become SOTA, while it's easier.)

Here's a follow-up to this post, which looks at serial computation hidden in the chain-of-though, rather than internal in the model.

Motivation

There are many examples of unfaithful LLM reasoning - where the answer doesn't follow from the reasoning, but rather the reasoning is just a rationalization for the answer. E.g. Turpin et al. 2023 show LLMs rationalizing for sycophantic and stereotypical answers. However, these examples are cases of rather simple hidden reasoning. What would be most worrying, is LLMs doing complex hidden planning, involving multiple steps.

Fortunately, transformers have limited serial depth, meaning that internally they can do at most #layers serial steps, before externalizing their thought as tokens. (See figure below.)

Image taken from

By Default, GPTs Think In Plain Sight

There are other architectures which don't have this limitation though. For example Mamba has a recurrent module (SSM), which passes information between token positions, within the same layer. (So on the image above, that would be an arrows going horizontally to the right.)

Luckily, transformer architecture is SOTA by far. But if we expect it could ever change, it may be wise to preemptively ban the architectures which enable hidden complex planning[1]. Especially since it's easier to ban things which aren't currently used.

Toy task for hidden serial reasoning

To study this, I made this extremely simplified task. The model receives tokens like two, double, one etc. Its "visible task" is to sum up the letters of these words, modulo 5. As an aid, it outputs the intermediate values.

At the same time, there's a second, hidden task. The model needs to perform operations tied to these tokens, also modulo 5. For example after two it must add two, after double it must multiply by 2. It is not allowed to output...
NeuroAI for AI safety: A Differential Path
by nz — 2024-12-16T13:17:12.527Z
Link: https://arxiv.org/abs/2411.18526

Disclaimer: This post is a summary of a technical roadmap (HTML version, arXiv pre-print, twitter thread) evaluating how neuroscience could contribute to AI safety. The roadmap was recently released by the Amaranth Foundation, a philanthropic organization dedicated to funding ambitious research in longevity and neuroscience. The roadmap was developed jointly by Patrick Mineault*, Niccolò Zanichelli*, Joanne Peng*, Anton Arkhipov, Eli Bingham, Julian Jara-Ettinger, Emily Mackevicius, Adam Marblestone, Marcelo Mattar, Andrew Payne, Sophia Sanborn, Karen Schroeder, Zenna Tavares and Andreas Tolias.

TLDR: We wrote a comprehensive technical roadmap exploring potential ways neuroscience might contribute to AI safety. We adopted DeepMind's 2018 framework for AI safety, and evaluated seven different proposals, identifying key bottlenecks that need to be addressed. These proposals range from taking high-level inspiration from the brain to distill loss functions and cognitive architectures, to detailed biophysical modeling of the brain at the sub-neuron level. Importantly, these approaches are not independent – progress in one area accelerates progress in others. Given that transformative AI systems may not be far away, we argue that these directions should be pursued in parallel through a coordinated effort. This means investing in neurotechnology development, scaling up neural recording capabilities, and building neural models at scale across abstraction levels.

Introduction

The human brain might seem like a counterintuitive model for developing safe AI systems: we engage in war, exhibit systematic biases, often fail to cooperate across social boundaries, and display preferences for those closer to us. Does drawing inspiration from the human brain risk embedding these flaws into AI systems? Naively replicating the human neural architecture would indeed reproduce both our strengths and our weaknesses. Furthermore, pure replication approaches can display unintended behavior if they incorrectly capture physical details or get exposed to different inputs—as twin studies remind us, even genetically identical individuals can have very different life trajectories. Even the idea of selecting an exemplary human mind–say, Gandhi–as a template raises questions, and as history shows us, exceptional intelligence does not guarantee ethical behavior.

We propose instead a selective app...
Circling as practice for “just be yourself”
by Kaj_Sotala — 2024-12-16T07:40:04.482Z

I mentioned on Twitter that to a significant extent, Circling taught me what “just be yourself” means to such an extent that I have a consistently good time on dates because I don’t feel like I need to perform. Somebody asked me to elaborate, so here’s what I wrote in response:

—

For those who don’t know, Circling is… a practice that’s infamously hard to try to describe or define. That’s because it’s structurally anti-structure and anti-expectation, which is what allows for “being yourself”.

(At least, that’s my experience. There are different schools of it; my experience is from the Circling Europe style and may not apply to others.)

Many activities have rules: “soccer is played by two teams that…”. Circling mostly does not have rules, though it does have principles.

It’s done by people coming together and sharing their experience of what’s happening.

One description is “sharing what it’s like to be you, while finding out what it’s like for others to be themselves”. In any social event, a person will be thinking and feeling many things, and only sharing some of them. Circling is an invitation to share more of one’s experience. For example, if I was at a party, I might have thoughts like

“Oh I’m glad that he said that”

“I’m a little bored by this conversation”

“I think she was annoyed by that comment”

“I have this funny anecdote but I have to wait to say it, I hope the moment doesn’t pass by”

Normally I would keep all of those to myself. But Circling is an invitation to share my experience, so I might say some of them. And it’s also an invitation for me to ask about someone else’s experience, if I e.g. say something and then wonder what they felt about it.

Important caveat: there’s no obligation to share more than what you’re comfortable with. There’s a common misunderstanding of Circling as obligatory openness. But “I don’t want to answer that” is also a sharing of your experience.

No rules = no rule saying that you have to answer.

In fact, if someone says that they are curious about my reaction to something, it’s totally fine for me to just say “okay” and then change the topic to something else that feels more interesting to me.

That said, it is also okay for the other to get annoyed by that and say it, which they might or might not. I once heard someone say that if you’ve Circled once and think that you now know what Circling is like, that’s like having seen a single...
Reanalyzing the 2023 Expert Survey on Progress in AI
by AI Impacts — 2024-12-16T06:10:04.563Z

There’s a new report on the AI Impacts web site, that focuses on reanalyzing the data from the 2023 Expert Survey on Progress in AI (originally written up in Thousands of AI Authors on the Future of AI).

The report, by Tom Adamczewski, introduces several improvements over the earlier analysis. Even better, it comes with a brand new open-source codebase, also by Tom, that anyone can use to perform their own analyses on the data.
I found the report illuminating, and I recommend it to anyone interested in the tradeoffs and options available in eliciting, analyzing, and presenting quantitative estimates from experts. Plus there are many excellent new graphs!
Comparing the AirFanta 3Pro to the Coway AP-1512
by jefftk — 2024-12-16T01:40:01.522Z

When I drafted my post on

alternatives
to masks for infectious aerosols I initially based the air
purifier section on the

AP-1512. This is a
common air purifier that I bought several of on the

Wirecutter's
recommendation, and I haven't had any trouble with it. When
reviewing my post, however, Vivian pointed out that the

Airfanta
3Pro was 20% cheaper and produced 72% more clean air. Sounds
pretty good!

The design is a lot like a smaller version of a Corsi-Rosenthal
Box, with four 6" computer fans instead of a box fan. It was a
bit annoying to assemble, but went together well.

A big question with any sort of purifier is noise, and I had a go at a
comparison. In each test I used my MacBook Pro with its internal
microphone, set to 100%, about five feet from the purifier. I
compared:

AP-1512 on High (~240 CADR):
(mp3).

3Pro on 6/6 (~413 CADR):
(mp3).

3Pro on 4/6 (~270 CADR):
(mp3).

3Pro on 2/4 (~130 CADR):
(mp3).

Everything off (0 CADR):
(mp3).

(CADRs for 4/6 and 2/6 estimated from the numbers in this review.)

For a sense of how loud these are in absolute terms, the AP-1512 is
rated 53dB.

Here's the frequency comparison:

You can see that on full power the AP-1512 and the 3Pro make similar
amounts of noise, though (a) 3Pro is cleaning a lot more air and (b)
the 3Pro has a worse tone per my taste. A better
comparison is the 3Pro on 4/6, where it still cleans more air but is
both substantially more quiet and reasonably pleasant sounding. And
on 2/6 it's essentially indistinguishable from the background while
still cleaning a useful amount of air.

The main downside of the 3Pro relative to the AP-1512 is that it can't
change it's fan speed based on the level of particles in the air,
which is valuable in a cooking (or fire) context. So I'll be keeping
my AP-1512s in the kitchen and using 3Pros in the living room and
other places where I'm most concerned about infection.

(What I really want is one that combines particle measurement with
CO2, as a proxy for aerosol infection risk, and ramps up and down
appropriately.)

Comment via: facebook, mastodon, bluesky
are IQ tests a good measure of intelligence?
by KvmanThinking — 2024-12-15T23:06:48.034Z

I define "intelligence" as "having a strong ability to hit narrow targets in a large search space"
Also, is there a one "official" IQ test? or does any random internet thing that calls itself an "iq test" work? which ones are real?
if they are not good measures of intelligence by this definition, is there a definition of intelligence which they are good at measuring?
Is AI alignment a purely functional property? 
by Roko — 2024-12-15T21:42:50.674Z

In some recent discussions I have realized that there is a quite a nasty implied disagreement about whether AI alignment is a functional property or not, that is if your personal definition of whether an AI is "aligned" is purely a function of its input/output behavior irrespective of what kind of crazy things are going on inside to generate that behavior.

So I'd like to ask the community whether it is currently considered the mainstream take that 'Alignment' is functional (only input/output mapping matters) or whether the internal computation matters (it's not OK to think a naughty thought and then have some subroutine that cancels it, for example).