trentmkelly/LessWrong-43k huggingface dataset:
If language is for communication, what does that imply about LLMs?
by Bill Benzon — 2024-05-12T02:55:39.724Z

Noam Chomsky famously believes that language originated to facilitate thought, but then came to be a medium of communication. Others believe the reverse, that it originated as a facility for communication which turned out to facilitate thinking. That is certainly my view.

If that is so, then one would think that language is structured to facilitate communication. Communication is serial, one token at a time, token after token. That would imply that language is structured to facilitate next-token prediction. That would in-turn imply that the relational structure of semantics would evolve to facilitate mapping between the linear structure of the language string and the multidimensional structure of meaning. You want to be able to efficiently project multidimensional semantic structure onto a string and to reconstruction multidimensional semantic structure from a string.

How are LLMs trained? By next token prediction. That is to say, the training regime mirrors the primary communication constraint governing the structure of language. So it is with text generation as well. Language is spoken one token at a time, and so LLMs generate texts, one token at a time.

The tasks that LLMs have trouble with, such as planning and arithmetic, ARE NOT primarily communicative in nature. They are tasks for thought, for reasoning.
[Linkpost] Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
by Bogdan Ionut Cirstea — 2024-05-11T22:59:12.368Z
Link: https://arxiv.org/abs/2405.04669

The excerpts below seem to me like a slight update towards arguments about the weakness of one-forward-passes in transformers and for agendas like externalized reasoning and translucent thoughts. They also suggest out-of-context reasoning (OOCR) might remain hard for transformer-based LMs (they currently do very poorly on OOCR evals and scaling also doesn't seem to help much):

We theoretically analyze reversal curse where training or test sequences have the from “𝐴→𝐵” or “𝐵←𝐴” via training dynamics of (stochastic) gradient descent under two auto-regressive models: a bilinear model (

Section 3

) and one-layer transformers under certain assumptions similar to Tian et al. (

2023a

) (

Section 4

). The analysis of the training dynamics of both models reveals a core reason why the reversal curse happens: the weights of the autoregressive models are asymmetric, i.e., the increase of weights from the token 𝐴 to token 𝐵 during training does not necessarily cause the increase of the weights from 𝐵 to 𝐴. [...] Although the (effective) weights from 𝐴 to 𝐵 and from 𝐵 to 𝐴 might be related to some extent since they are both computed using the same set of embeddings, their correlation is weak and thus show asymmetry as verified both theoretically (

Sections 3

and

4

) and empirically (

Section 5

).

we use the above framework to show the necessity of chain-of-thought (COT) (Wei et al.,

2022b

) (i.e., a model trained on “A implies B” and “B implies C” separately struggles to directly conclude “A implies C” without COT, which was also empirically observed by Allen-Zhu and Li (

2023

)) via training dynamics of one-layer transformers (

Section 4.2

), which provides a new perspective different from previous work Feng et al. (

2024

) that focuses on the expressivity of transformers. Slightly different from the reason for the reversal curse, in COT analysis, the model weights show intransitivity, i.e., increasing the weights from the token 𝐴 to 𝐵 and 𝐵 to 𝐶 does not necessarily increase the weights from 𝐴 to 𝐶. We emphasize again that the weights refer to effective weights.

We also empirically validate our theoretical results through the training dynamics of multi-layer transformers (

Section 5

).

The asymmetry and intransitivity of weights of auto-regressive models indicate that auto-regressive LLMs might not auto

...
How To Do Patching Fast
by Joseph Miller — 2024-05-11T20:13:52.424Z

This post outlines an efficient implementation of Edge Patching that massively outperforms common hook-based implementations. This implementation is available to use in my new library, AutoCircuit, and was first introduced by Li et al. (2023).

What is activation patching?

I introduce new terminology to clarify the distinction between different types of activation patching.

Node Patching

Node Patching (aka. “normal” activation patching) is when some activation in a neural network is altered from the value computed by the network to some other value. For example we could run two different prompts through a language model and replace the output of Attn 1 when the model is given some input 1 with the output of the head when the model is given some other input 2.

We will use the running example of a tiny, 1-layer transformer, but this approach generalizes to any transformer and any residual network.

All the nodes downstream of Attn 1 will be affected by the patch.

Edge Patching

If we want to make a more precise intervention, we can think about the transformer differently, to isolate the interactions between components.

Now we can patch the edge Attn 1 -> MLP and only nodes downstream of MLP will be affected (eg. Attn 1->Output is unchanged). Edge Patching has not been explicitly named in any prior work.

Path Patching

Path Patching refers to the intervention where an input to a path is replaced in the ‘treeified’ view of the model. The treeified view is a third way of thinking about the model where we separate each path from input to output. We can implement an equivalent intervention to the previous diagram as follows:

In the IOI paper, ‘Path Patching’ the edge Component 1 -> Component 2 means Path Patching all paths of the form

Input -> ... -> Component 1 -> ... -> Component 2 -> ... -> Output

where all components between Component 1 and Component 2 are MLPs[1]. However, it can be easy to confuse Edge Patching and Path Patching because if we instead patch all paths of the form

Input -> ... -> Component 1 -> Component 2 -> ... -> Output

this is equivalent to Edge Patching the edge Component 1->Component 2.

Edge Patching all of the edges which have some node as source is equivalent to Node Patching that node. AutoCircuit does not implement Path Patching, which is much more expensive in g...
Can we build a better Public Doublecrux?
by Raemon — 2024-05-11T19:21:53.326Z

Something I'd like to try at LessOnline is to somehow iterate on the "Public Doublecrux" format. I'm not sure if I'll end up focusing on it, but here are some ideas.

Public Doublecrux is a more truthseeking oriented version of Public Debate. The goal of a debate is to change your opponent's mind or the public's mind. The goal of a doublecrux is more like "work with your partner to figure out if you should change your mind, and vice versa."

Reasons to want to do public doublecrux include:

It helps showcase subtle mental moves that are hard to write down explicitly (i.e. tacit knowledge transfer.
There's still something good and exciting about seeing high profile smart people talk about ideas. Having some variant of that format seems good for LessOnline. And having at least 1-2 "doublecruxes" rather than "debates" or "panels" or "interviews" seems good for culture setting.
In addition to being "exciting" and "possible to learn from" to have public figures doublecrux, I think it'd also be nice from a culture setting standpoint. This is a place where people don't play rhetorical tricks to manipulate people – it's a place where people earnestly move towards the truth.

Sidebar: Public Debate is also good although not what I'm gonna focus on here.

I know several people who have argued that "debate-qua-debate" is also an important part of a truthseeking culture. It's fine if the individuals are trying to "present the best case for their position", so long as the collective process steers towards truth. Adversarial Collaboration is good. Public disagreement is good.

I do generally buy this, although I have some disagreements with the people who argue most strongly for Debate. I think I prefer it to happen in written longform than in person, where charisma puts a heavier thumb on the scale. And I think while it can produce social good, many variants of it seem... kinda bad for the epistemic souls of the people participating? By becoming a champion for a particular idea, people seem to get more tunnel-vision-y about it. Sometimes worth it, but, I've felt some kind of missing mood here when arguing with people in the past.

I'm happy to chat about this in the comments more but mostly won't be focusing on it here.

Historically I think public doublecruxes have had some problems:

First, having the live audience there makes it a bit more awkward and performative. It's harder to "

...
Resources for learning about poise / gracefulness?
by David Gross — 2024-05-11T18:30:32.428Z

I'm doing some initial investigation for a Notes on Virtues sequence write-up about poise & gracefulness (possibly touching also on related things like confidence, composure, cool, gravitas, command, bearing, etc.) seen as a virtue (a characteristic habit that promotes or exhibits human flourishing).

Can you recommend some resources (e.g. books, papers, interviews) I would be wise to review for this? I'm especially interested in things that people can do to improve in this virtue; interventions that have some scientifically demonstrated support being ideal, but informal advice from exemplars and experts are also valuable.
New intro textbook on AIXI
by Alex_Altair — 2024-05-11T18:18:50.945Z

Marcus Hutter and his PhD students David Quarel and Elliot Catt have just published a new textbook called An Introduction to Universal Artificial Intelligence.

"Universal AI" refers to the body of theory surrounding Hutter's AIXI, which is a model of ideal agency combining Solomonoff induction and reinforcement learning. Hutter has previously published a book-length exposition of AIXI in 2005, called just Universal Artificial Intelligence, and first introduced AIXI in a 2000 paper. I think UAI is well-written and organized, but it's certainly very dense. An introductory textbook is a welcome addition to the canon.

I doubt IUAI will contain any novel results, though from the table of contents, it looks like it will incorporate some of the further research that has been done since his 2005 book. As is common, the textbook is partly based on his experiences teaching the material to students over many years, and is aimed at advanced undergraduates.

I'm excited for this! Like any rationalist, I have plenty of opinions about problems with AIXI (it's not embedded, RL is the wrong frame for agents, etc) but as an agent foundations researcher, I think progress on foundational theory is critical for AI safety.

Basic info

Hutter's website
Releasing on May 28th 2024
Available in hardcover, paperback and ebook
496 pages

Table of contents:

Part I: Introduction

1. Introduction

2. Background

Part II: Algorithmic Prediction

3. Bayesian Sequence Prediction

4. The Context Tree Weighting Algorithm

5. Variations on CTW

Part III: A Family of Universal Agents

6. Agency

7. Universal Artificial Intelligence

8. Optimality of Universal Agents

9. Other Universal Agents

10. Multi-agent Setting

Part IV: Approximating Universal Agents

11. AIXI-MDP

12. Monte-Carlo AIXI with Context Tree Weighting

13. Computational Aspects

Part V: Alternative Approaches

14. Feature Reinforcement Learning

Part VI: Safety and Discussion

15. AGI Safety

16. Philosophy of AI
Questions are usually too cheap
by Nathan Young — 2024-05-11T13:00:54.302Z
Link: https://nathanpmyoung.substack.com/p/questions-are-too-cheap

It is easier to ask than to answer.

That’s my whole point.

It is much cheaper to ask questions than answer them so beware of situations where it is implied that asking and answering are equal.

Here are some examples:

Let's say there is a maths game. I get a minute to ask questions. You get a minute to answer them. If you answer them all correctly, you win, if not, I do. Who will win?

Preregister your answer.

Okay, let's try. These questions took me roughly a minute to come up with.

What's 56,789 * 45,387?

What's the integral from -6 to 5π of sin(x cos^2(x))/tan(x^9) dx?

What's the prime factorisation of 91435293173907507525437560876902107167279548147799415693153?

Good luck. If I understand correctly, that last one's gonna take you at least an hour1 (or however long it takes to threaten me).

Perhaps you hate maths. Let's do word problems then.

Define the following words "antidisestablishmentarianism", "equatorial", "sanguine", "sanguinary", "escapology", "eschatology", "antideluvian", "cripuscular", "red", "meter", all the meanings of "do", and "fish".

I don’t think anyone could do this without assistance. I tried it with Claude, which plausibly still failed2 the “fish” question, though we’ll return to that.

I could do this for almost anything:

Questions on any topic

Certain types of procedural puzzles

Asking for complicated explanations (we’ll revisit later)

Forecasting questions

This is the centre of my argument

I see many situations where questions and answers are treated as symmetric. This is rarely the case. Instead, it is much more expensive to answer than to ask.

Let’s try and find some counter examples. A calculator can solve allowable questions faster than you can type them in. A dictionary can provide allowable definitions faster than you can look them up. An LLM can sometimes answer some types of questions more cheaply in terms of inference costs than your time was worth in coming up with them.

But then I just have to ask different questions. Calculators and dictionaries are often limited. And even the best calculation programs can’t solve prime factorisation questions more cheaply than I can write them. Likewise I could create LLM prompts that are very expensive for the best LLMs to answer well, eg “write a 10,000 word story about an [animal] who experiences [emotion] in a [location].”

How this plays out

Let’s go back...
Ethics and prospects of AI related jobs?
by dr_s — 2024-05-11T09:31:04.190Z

I've been on the lookout for new jobs recently and one thing I have noticed is that the market seems flooded with ads for AI-related jobs. What I mean is not work on building models (or aligning them, alas), but rather, work on building applications using generative AI or other advances to make new software products.
My impression of this is that first, there's probably something of a bubble, because I doubt many of these ideas can deliver on their promises, especially as they rely so heavily on still pretty unreliable LLMs and such. And second, that while the jobs are well paid and sound fun, I'm not sure how I feel about them. These jobs all essentially aim at automating away other jobs, one way or another. That is a good thing only insofar as various other things happen, and depending on the specific job and quality of the work - a good automated GP for diagnosis would probably do a lot of good, but a rushed one might be net negative, and automating creative work is IMO just the wrong road to go down in general if we want good AI futures.

What are your intuitions about this? Which kinds of AI jobs do you consider having more potential for overall positive/negative value for society?
Should I Finish My Bachelor's Degree?
by Zack_M_Davis — 2024-05-11T05:17:40.067Z
Link: http://zackmdavis.net/blog/2024/05/should-i-finish-my-bachelors-degree/

To some, it might seem like a strange question. If you think of being college-educated as a marker of class (or personhood), the fact that I don't have a degree at age of thirty-six (!!) probably looks like a scandalous anomaly, which it would be only natural for me to want to remediate at the earliest opportunity.

I deeply resent that entire worldview—not because I've rejected education, properly understood. On the contrary. The study of literature, history, mathematics, science—these things are among the noblest pursuits in life, sources of highest pleasure and deepest meaning. It's precisely because I value education so much that I can't stand to see it conflated with school and its culture of bureaucratic servitude where no one cares what you know and no one cares what you can do; they just want you to sit in a room and obey the commands of the designated teacher. Whereas in reality, knowledge doesn't come from "taking courses."

How could it? Knowledge comes from quality study and practice. Sure, it's possible that someone could study in order to "pass" a "class" that they're "taking" in school. But once you know how and why to study, it's not clear what value the school is adding that can't be gotten better, cheaper, elsewhere. Just get the books. (And start a blog, go to meetups, chat to large language models, hire a private tutor—whatever makes sense to get better at doing the things you want to do, without having to worry about whether the thing that makes sense can be made legible to distant bureaucrats.)

The people who believe in being college-educated probably don't believe me. They probably think my pæans to the glory of self-study are the rationalizations of a lazy student who doesn't want to work hard.

I can understand some reasons for skepticism. Sometimes people really are lazy, and suffer from self-serving delusions. Probably there are some confused people out there who have mistaken consumer edutainment for production scholarship and—maybe, somehow—could benefit from being set straight by the firm tutelage of the standard bureaucratic authority.

But without vouching for everyone who calls themself an autodidact, I think I can present third-party-visible evidence that my self-study is for real? I worked as a software engineer for eight years; I have 173 commits in the Rust compiler; I wrote a chess engine; I've blogged 400,000 words over the past...
Custom Audio Switch Box
by jefftk — 2024-05-11T02:40:02.031Z

When I play live I have a bunch of instruments, including:

Mandolin: an electric mandolin

Computer: a custom MIDI mapper driven by keyboard, foot drums, and
breath controller.

Bass whistle: a whistle-controlled
bass synthesizer

I also have some effects, primarily a talkbox
and an audio-to-audio synth
pedal. Normally I route the mandolin into the effects, but I've
recently been wanting more options:

The computer effects are a lot of fun routed through the
talkbox.

If I set the bass whistle to emit just a sine wave, and pipe
that into the synth pedal, I can control professionally-designed
sounds:

The thing that makes this tricky is that I want to be able to play
mandolin direct (which goes via the talkbox output) at the same time
as playing bass whistle (which goes via the pedals output). I
sketched a lot of options:

And eventually realized I only needed something simple:

This lets me switch between two modes:

Mandolin goes to talkbox, whistle goes to pedals.

Mandolin goes to pedals, computer goes to talkbox.

I wired something up with a 3PDT switch, five 1/4" jacks, and a
couple wagos, in a a
small case:

The 1/4" jacks are very slightly too big for the case, so I needed to
make them very slightly smaller. Which I did (not recommended) by
melting away the offending region with the soldering iron.

I tested it with my mandolin pretending to be each instrument, and it
seems to work! The main risk is that the volumes of the input
instruments aren't in the right range, which I won't know until I
combine all three instruments. If that's the situation I think I can add some
resistors to balance it out.

If the levels are decent, though, I'll give it a go at Porchfest
tomorrow!

Comment via: facebook, mastodon
MATS Winter 2023-24 Retrospective
by utilistrutil — 2024-05-11T00:09:17.059Z

Co-Authors: @Rocket, @LauraVaughan, @McKennaFitzgerald, @Christian Smith, @Juan Gil, @Henry Sleight, @Matthew Wearden, @Ryan Kidd

The ML Alignment & Theory Scholars program (MATS) is an education and research mentorship program for researchers entering the field of AI safety. This winter, we held the fifth iteration of the MATS program, in which 63 scholars received mentorship from 20 research mentors. In this post, we motivate and explain the elements of the program, evaluate our impact, and identify areas for improving future programs.

Summary

Key details about the Winter Program:

The four main changes we made after our Summer program were:

Reducing our scholar stipend from $40/h to $30/h based on alumni feedback;
Transitioning Scholar Support to Research Management;
Using the full Lighthaven campus for office space as well as housing;
Replacing Alignment 201 with AI Strategy Discussions.

Educational attainment of MATS scholars:

48% of scholars were pursuing a bachelor’s degree, master’s degree, or PhD;
17% of scholars had a master’s degree as their highest level of education;
10% of scholars had a PhD.

If not for MATS, scholars might have spent their counterfactual winters on the following pursuits (multiple responses allowed):

Conducting independent alignment research without mentor (24%);
Working at a non-alignment tech company (21%);
Conducting independent alignment research with a mentor (13%);
Taking classes (13%).

Key takeaways from scholar impact evaluation:

Scholars are highly likely to recommend MATS to a friend or colleague (average likelihood is 9.2/10 and NPS is +74).
Scholars rated the mentorship they received highly (average rating is 8.1/10).

For 38% of scholars, mentorship was the most valuable element of MATS.

Scholars are likely to recommend Research Management to future scholars (average likelihood is 7.9/10 and NPS is +23).

The median scholar valued Research Management at $1000.
The median scholar reported accomplishing 10% more at MATS because of Research Management and gaining 10 productive hours.

The median scholar made 5 professional connections and found 5 potential future collaborators during MATS.
The average scholar self-assessed their improvement on the depth of their technical skills by +1.53/10, their breadth of knowledge by +1.93/10, their research taste by +1.35/10, and their theory of change constru

...