trentmkelly/LessWrong-43k huggingface dataset:
Grammars, subgrammars, and combinatorics of generalization in transformers
by Dmitry Vaintrob — 2025-01-02T09:37:23.191Z

Introduction

This is the first installment of my January writing project. We will look at generative neural networks from the framework of (probabilistic) "formal grammars", specifically focusing on building a complex grammar out of simple “rule grammars”. This turns out to lead to a nice, and relatively non-technical way of discussing how complex systems like language models can be built out of "heuristics". Thinking more about how these blocks are combined (and focussing on the difference between combining rules via "AND" vs. "OR") leads to some new insights on generalization, which are sometimes lost in the fuzzy language of heuristics and circuits. In a follow-up to this post we'll apply the tools introduced here in a more concrete context, of the recent paper on an in-context variant of modular addition, which provides a particularly nice example of the distinctions inherent in the "subgrammar" picture. I'll also sketch some preliminary results from ongoing research about how neural networks might learn through analogy, a mechanism that dovetails in an interesting way with the formal grammar discussions here, but implies a new and interesting class of mechanisms that goes beyond these ideas.

Grammars, subgrammars, and circuits

A grammar (also known as “language” or “syntax”) is an “accept/reject” machine on strings. It accepts a finite string s of tokens (these can be bits, words, letters, numbers, LLM tokens, etc.) and outputs a “grammar check” G(s)∈{0,1} which is “True” if the string is grammatical and “False” if it is not.
Technically, any boolean classifier on strings is a grammar. But in practice, the term “grammar” denotes a particular way of generating and analyzing such classifiers. In particular, this term tends to be used for classifiers which decompose into formal and (in a suitable sense) local rules. We’ll revisit all of this later (focusing in particular on locality). But to keep things concrete, let’s introduce a toy grammar that we’ll be carrying around as a simple example. We define G_0 to be the grammar on all lowercase English letters, a-z, which accepts a string if and only if it satisfies the following two rules:

R1: “q” must be followed by “u”

R2: “i” before “e” except after “c”. In other words, this grammar rejects the substring “ei” unless it is part of the substring “cei”.
Note that this grammar is particularly simple (in parti

...
Grading my 2024 AI predictions
by Nikola Jurkovic — 2025-01-02T05:01:46.587Z

On Jan 8 2024, I wrote a Google doc with my AI predictions for the next 6 years (and slightly edited the doc on Feb 24). I’ve now quickly sorted each prediction into Correct, Incorrect, and Unclear. The following post includes all of my predictions for 2024 with the original text mostly unedited and commentary in indented bullets.

Correct

there is a viral app (probably Suno) for generating music, reaching 1 million users by July 2024

Suno had 10 million users in May.

An open source GPT-4-level model is released.

Llama 3.1

probably fits the bill.

Adept AI and similar publicly available browser-based assistants are still not useful enough to be used on browser windows without being supervised by a human for more than 30 seconds. They still have problems like clicking on the wrong part of the screen, getting lost, getting distracted, etc.

I haven’t seen any agents that are actually able to navigate a browser competently yet.

Sora is released to customers who apply for access.
If OAI makes the video continuation feature available, many new memes are created where people use Sora to extend existing videos in funny ways or stitch two videos together.

Example

(although these don’t use Sora). I find it amusing how specific this prediction was. Possibly I’d already seen an example at that point?

We will see the first signs of evals for moral patienthood in LLMs. Some of the AGI labs will make a public statement where they mention this possibility.

The Anthropic Fellows Program

is looking for people to work on “AI Welfare: Improving our understanding of potential AI welfare and developing related evaluations and mitigations.”

6/12 METR tasks are complete

This suite is deprecated but my best guess is that it would resolve Correct.

1/5 ARA tasks are complete

This suite (the five tasks described in Anthropic's original RSP) is deprecated but it’s likely true that Claude 3.5 Sonnet Upgraded would complete at least one task.

Incorrect

AI music leads to protests/complaints in the music industry. Major artists (>3% of playtime-weighed artists) say something like “AI music is bad!”.
Microsoft Copilot and similar tools increase office worker productivity (speed on <1hr tasks) by 25%. Most of the accelerated labor is pretty menial (making presentations, writing emails, making/manipulating spreadsheets)
OpenAI keeps some sort of logs for more than half the videos it generates, so that it can

...
Practicing Bayesian Epistemology with "Two Boys" Probability Puzzles
by Liron — 2025-01-02T04:42:20.362Z

The Puzzles

There's a simple Monty Hall adjacent probability puzzle that goes like this:

Puzzle 1
I have two children, at least one of whom is a boy. What is the probability that both children are boys?

A more complex variation recently went viral on Twitter:

Puzzle 2
I have two children, (at least) one of whom is a boy born on a Tuesday — what is the probability that both children are boys?

Then Isaac King tweeted an even more complex variation:

Puzzle 3
I have two children, at least one of whom is a boy born on a day that I'll tell you in 5 minutes. What is the chance that both are boys, and what will the chance be after I tell you the day?

All three versions are fun and worth a try if you want to learn and practice Bayesian reasoning.

Personally, I found Isaac's version MUCH harder than the others. I was surprised how hard it stumped me since I had a pretty easy time with the first two (LessWrong trained me well). As I stared at it for longer than I want to admit, the gears of my brain kept jamming. I couldn't see a coherent non-paradoxical story for what the Bayesian updates should look like.

I recommend giving Puzzle 3 a try before reading my solution.

By the way, I tested it on GPT-o1 and Claude 3.5 Sonnet and they only give incorrectly-reasoned wrong answers. It'll be interesting to see if o3 can do better.

The Solution

First, when we hear the "I have two children, at least one of whom is a boy" part, we set the probability of two boys to 1/3 because the possibilities {(boy, girl), (girl, boy), (boy, boy)} are a-priori equally likely and we haven't had a reason to update their relative likelihoods.

Then when we hear "I'll tell you the day that at least one was born on", we don't need to update the three relative likelihoods because it's a statement we were equally likely to hear in all three possible worlds.

Now the tricky part…

When we subsequently hear a particular day, e.g. "Friday", how should we update the relative probabilities?

It seems like we shouldn't update, because hearing any weekday was a-priori equally likely to hear, and if hearing any weekday was going to update us in any particular direction, why couldn't we have just made that update before we heard the particular day?

In other words, why wouldn't we pretend like the parent mumbled the day and we couldn't make out the word, but update anyway, since it's going to be the same update regardless of w...
Implications of Moral Realism on AI Safety
by Myles H — 2025-01-02T02:58:42.527Z

Epistemic Status: Still in a brainstorming phase - very open to constructive criticism.

I'll start by clarifying my definition of moral realism. To begin with an example, here is what a moral realist and anti-realist might say on the topic of suffering:

Moral Realist: The suffering of sentient beings is objectively wrong therefore I want to minimize it

Moral Anti-Realist: I want to minimize the suffering of sentient beings

Moral realists have justifiable terminal goals. They reject the notion that is and ought statements can't mix. A moral realist says that some ought statements fall into the is category, and those that don't are invalid.

A moral realist looks outward to their environment to discover what they should want where an anti-realist looks inward and asks themselves what they want.

A moral realist can make statements like, "It is correct to want X, and incorrect to want Y." Thus, they would expect any perfectly rational agent to only pursue goals that are true.

By (my) definition of moral realism, the orthogonality thesis is false, or certainly not as strong as typically described.

Omnizoid has a great post on the topic - The Orthogonality Thesis is Not Obviously True. The post already very thoughtfully argues the position so instead I will focus more on its implications for approaching AI safety.

The most popular technical approach to AI safety is AI alignment, often described as follows: Develop techniques to ensure AI robustly pursues any goal a user provides without causing unintended net-negative consequences according to the user's preferences.

The hope is that we can then provide this loyal AI with goals humans collectively want, and enact laws and regulations to ensure bad actors don't give the AI bad goals.

If moral realism is true then this is a bad and totally intractable approach to AI safety.

Under this agenda, one tries to make it possible to instill an AI with any arbitrary goal, including those that aren't valid. For one, this then puts the burden on humans to figure out what is objectively good. Secondly, it unnecessarily goes out of its way to make instilling immoral objectives possible. Lastly, I have no idea how you get around instrumental convergence. A highly intelligent arbitrarily aligned AI has profound economic utility, but it is not a moral pursuit.

Instead, I propose a two pronged approach to developing ASI (artificial super intellig...
Read The Sequences As If They Were Written Today
by Peter Berggren — 2025-01-02T02:51:36.537Z

If you've never read the LessWrong Sequences (which I read through the book-length compilation Rationality: From AI To Zombies), I suggest that you read the Sequences as if they were written today. Additionally, if you're thinking of rereading the Sequences, I suggest that your agenda for rereading, in addition to what it may already be, should be to read the Sequences as if they were written today.

To start, I'd like to take a moment to clarify what I mean. I don't mean "think about what you remember the Sequences talking about, and try to apply those concepts to current events." I don't even mean "read the Sequences and reflect on where the concepts are relevant to things that have happened since they were written." What I mean is that you should read the Sequences as if they were written today. You should imagine that, on January 1, 2025 (or whenever you happen to be reading this post), whatever post you're reading has just been released by some unknown 20-something-year-old alignment researcher, you saw it while scrolling down a bit on the front page of LessWrong, the title caught your eye, and you started reading it.

Key advantages of this approach

1. It's easier to notice the contemporary applicability of the Sequences if you think of them as contemporary

Many Sequences posts seem to be responding very directly to certain historical events within the rationalist community and Internet culture more broadly, at least if you approach them as a historical document. If you read them as if they were written today, however, they will seem instead to be responding very directly to certain current events within the rationalist community and Internet culture more broadly.

Strictly speaking, I don't think either one of these understandings is accurate, especially given how heavily they were based on scholarly work from decades earlier (e.g. the work of Tversky and Kahneman) that was likely in the works for a long time before it was even published. However, it's natural to try and apply them to whenever they were written, given how much Eliezer Yudkowsky is idolized (however unintentionally) as an original thinker and how comparatively uncommon it is to fully account for the giants whose shoulders he stood on.

While it may be intellectually interesting to see how Yudkowsky's work applies to the time in which it was written, there are a few key weaknesses to ...
A Collection of Empirical Frames about Language Models
by Daniel Tan — 2025-01-02T02:49:05.965Z

What's the sum total of everything we know about language models? At the object level, probably way too much for any one person (not named Gwern) to understand.

However, it might be possible to abstract most of our knowledge into pithily-worded frames (i.e. intuitions, ideas, theories) that are much more tractable to grok. And once we have all this information neatly written down in one place, unexpected connections may start to pop up.

This post contains a collection of frames about models that are (i) empirically justified and (ii) seem to tell us something useful. (They are highly filtered by my experience and taste.) In each case I've distilled the key idea down to 1-2 sentences and provided a link to the original source. I've also included open questions for which I am not aware of conclusive evidence.

I'm hoping that by doing this, I'll make some sort of progress towards "prosaic interpretability" (final name pending). In the event that I don't, having an encyclopedia like this seems useful regardless.

I'll broadly split the frames into representational and functional frames. Representational frames look 'inside' the model, at its subcomponents, in order to make claims about what the model is doing. Functional frames look 'outside' the model, at its relationships with other entities (e.g. data distribution, learning objectives etc) in order to make claims about the model.

---

This is intended to be a living document; I will update this in the future as I gather more frames. I strongly welcome all suggestions that could expand the list here!

Things we're interested in understanding

Obviously it'd be nice to understand "language model behaviour" generally, but we seem far away from this. Specific things might be more tractable to understand in isolation.

Refusal
In-context learning
Reasoning (e.g. through chain of thought)
Memorization (i.e. factual recall)
Models' assumed persona / identity
Self-awareness (a.k.a situational awareness, introspection)

Representational Frames

Transformer computation can be broken down into nearly-linear 'circuits', which in turn explain how they compute simple bigrams / trigrams.
Transformers near-universally contain 'induction heads' that detect / modulate repetitive sequences.
Transformers represent features in superposition as almost-orthogonal directions, of which there can be exponenti

...
My January alignment theory Nanowrimo
by Dmitry Vaintrob — 2025-01-02T00:07:24.050Z

Update: list of posts so far. "(s)" denotes shortform.

post 1, post 2, post 3, post 4 (s), post 5 (s), post 6 (s), post 7, post 8, post 9 (s), post 10, post 11, post 12, post 13 (s), post 14, post 15, post 16 (with Lauren Greenspan), post 17, post 18, post 19 (s), post 20, post 21.

*****

This is a quick announcement/commitment post:

I've been working at the PIBBSS Horizon Scanning team (with Lauren Greenspan and Lucas Teixeira), where we have been working on reviewing some "basic-science-flavored" alignment and interpretability research and doing talent scouting (see this intro doc we wrote so far, which we split off from an unfinished larger review). I have also been working on my own research. Aside from active projects, I've accumulated a bit of a backlog of technical writeups and shortforms in draft or "slack discussion"-level form, with various levels of publishability.

This January, I'm planning to edit and publish some of these drafts as posts and shortforms on LW/the alignment forum. To keep myself accountable, I'm committing to publish at least 3 posts per week.

I'm planning to post about (a subset? superset? overlapping set? of) the following themes:

Opinionated takes on a few research directions (I have drafts on polytopes, mode connectivity, and takes on proof vs. other kinds of "principled formalism without proofs").
Notes on grammars and more generally, how simpler rules and formal structures can combine into larger ones. This overlaps with a project I'm working on with collaborators, involving a notion of "analogistic circuits": mechanisms that learn to generalize a complex rule "by analogy", without ever encoding the structure itself.
Joint with Lauren Greenspan and Lucas Teixeira: some additional bits of our review, with a focus on interepretability (and ways to think about assumptions and experiments).
Joint with Lauren: some distillation and discussion of QFT methods in interpretability.
Bayesian vs. SGD learning from various points of view. (Closely related to discussions with Kaarel Hänni, Lucius Bushnaq, and others).
Related to the above: Extensions of the "Low-Hanging-Fruit" prior post with Nina Panicksserry, specifically focusing on non-learnability of parity, and a new notion of "training stories" (this is closely related to some other work we've done with Nina, as well as joint work with Louis Jaburi).
???

I am generally resistant to ...
Intranasal mRNA Vaccines?
by J Bostock — 2025-01-01T23:46:40.524Z

This is not advice. Do not actually make this, and especially do not make this and then publicly say "I snorted mRNA because Jonathan said it was a good idea". Because I'm not saying it's a good idea.

Everyone remembers johnswentworth making RaDVac almost four years ago now. RaDVac was designed to be, well, rapidly deployed, so it uses short peptides, rather than longer peptides or inactivated virus, which are what normal vaccines use.

Since then, we've seen the introduction of mRNA vaccines, which can also be used intranasally! So would it be possible to produce something like this at home?

The Non-mRNA Components

mRNA vaccines consist of various. The first is the mRNA itself, the other components are a bunch of lipids (read: fatty molecules) which form into tiny particles rather unimaginatively called lipid nanoparticles (LNPs). These consist of a bunch of lipids surrounding the mRNA. Their job is to stick to cells, and then kind of merge with the cell's membrane (like two bubbles popping together into one big bubble) and release the mRNA into the cell. This works because the LNPs are cationic (positively charged) and cell membranes tend to be negatively charged.

There are sometimes other steps wherein the LNPs are actively taken up, transferred to an internal compartment, and then break out of that compartment.

So my first guess was to just buy something called Lipofectamine:

In this hypothetical case, we'd ignore steps 1 and 7, and replace step 6 with "huff it".

(Side note: "70-90% confluent" just means that the slides are 70-90% covered in cells, it has nothing to do with any property of the cells themselves, which is why we won't worry about it.)

The question is, would this work? Lipofectamine is probably similar to the lipid composition of the LNPs from this paper but not the same. I spoke to a friend whose job is getting nucleic acids into lung cells (lung cells and nasal cells are relatively similar) and (paraphrased) she said "Don't DIY an mRNA vaccine" but then she said "Uptake rates for [those kinds of cells] are usually low ... but mRNA is easier to get into cells than what I work with".

So it's unclear whether lipofectamine as bought would work. There are lots of different lipofectamine formulations, but I can't at a glance tell which one would be best. Depending on the amount of this you want, it could be from $100 to $1000.

The mRNA

Our bigge...
Example of GPU-accelerated scientific computing with PyTorch
by Tahp — 2025-01-01T23:01:04.606Z
Link: https://passwordpaper.com/PyTorch-GPU-API/

Here's a fun little post I made because a friend asked me how PyTorch had things which were supported in the CUDA backend but not the MPS backend. I was once the sort of person who was on LessWrong, would find the subject interesting, and not already know everything in the post, so I'm posting it here to see if there's enough interest to post stuff like this here in the future. I'm worried that the average person reading LessWrong will find this post too elementary, but I'm trying to post more so I can practice writing before I have something interesting to say. Feedback appreciated.

As part of my physics research, I've been using some C++ code which performs quantum mechanics calculations. I need to do the same set of linear algebra calculations millions of times with independently generated random inputs, and it occurred to me that this sounded like the sort of thing GPUs would be good at. GPUs are designed to perform a lot of simple calculations at the same time. My old code is made to run on CPUs which are good doing a lot of complicated calculations one after another. For those who are unfamiliar, the recent AI revolution is largely the product of researchers turning a prediction algorithm based on linear algebra into many (relatively) smaller calculations that may all be run at the same time by a GPU. If you know how to multiply matrices and want some flavor for how things like ChatGPT are just linear algebra, this page seemed correct to me as of a quick read on Jan 1, 2025. Anyway, the important thing is that I knew that a bunch of engineers had put a lot of effort into making many large linear algebra calculations run quickly at the same time on GPUs, and it seemed a shame not to use that for physics. I've been itching to rewrite my research code to take advantage of modern machine learning tools for months, and I finally started doing it a few weeks ago. I decided to use PyTorch, because I have access to a lot of tools which make rapidly testing Python code easier than testing C++ code, and the research computers at my university already have Torch installed. Python is much much slower than C++ code, but that shouldn't matter for reasons I explain below.

So far it seems like it's working! I think I've figured out how to turn my calculation into operations that PyTorch knows how to batch together, but I need to fix the mathematical details of the code....
Economic Post-ASI Transition
by Joel Burget — 2025-01-01T22:37:31.722Z

Who's done high quality work / can tell a convincing story about managing the economic transition to a world where machines can do every job better than humans?

Some common tropes and why I don't think they're good enough:

"We've always managed in the past. Take the industrial revolution for example. People stop doing the work that's been automated and find new, usually better-compensated work to do." This is true, and I think it will probably be an important component of the transition. But it's clearly not sufficient if machines are better than humans at everything.
"Tax AI (developers?) to pay for UBI." Again, something in this vein will probably be part of the solution. But:

(a) UBI hasn't been well-tested.
(b) I don't think the math works out if / when AI companies dominate the economy, since they'll capture more and more of the economy unless tax rates are high enough that everyone else receives more through UBI than they're paying the AI companies.
(c) It doesn't have enough detail.

Worldcoin. I think the idea is similar to the UBI story, but again it needs more detail.

Who has thought about this really deeply / well?

Note that for the purpose of this question, assume a world where alignment basically works (we can debate that question elsewhere).