trentmkelly/LessWrong-43k huggingface dataset:
Understanding LLMs: Some basic observations about words, syntax, and discourse [w/ a conjecture about grokking] 
by Bill Benzon — 2023-10-11T19:13:30.432Z

Cross posted from New Savanna.

I seem to be in the process of figuring out what I’ve learned about Large Language Models in the process of playing around with ChatGPT since December of last year. I’ve already written three posts during this phase, which I’ll call my Entanglement phase, since this re-thinking started with the idea that entanglement is the appropriate way to think about word meaning in LLMs. This post has three sections.

The first section is stuff from Linguistics 101 about form and meaning in language. The second argues that LLMs are an elaborate structure of relational meaning between words and higher order structures. The third is about the distinction between sentences and higher-level structures and the significance that has for learning. I conjecture that there will come point during training when the engine learns to make that distinction consistently and that that point will lead to a phase change – grokking? – in its behavior.

Language: Form and Meaning

Let us start with basics: Linguists talk of form and meaning; Saussure talked of signifier and signified. That is to say, words consist of a form, or signifier, a physical signal such as a sound or a visual image, which is linked to or associated with a meaning, or signified, which is not so readily characterized and, in any event, is to be distinguished from the referent or interpretant (to use Pierce’s term). Whatever meaning is, it is something that exists in the minds/brains of speakers and only there.

Large Language Models are constructed over collections of linguistic forms or signifiers. When humans read texts generated by LLMs, we supply those strings of forms with meanings. Does the LLM itself contain meanings? That’s a tricky question.

On one sort of account, favored by at least some linguistics and others, no, they do not contain meanings. On a different sort of account, yes, they do. For the LLM is a sophisticated and complicated structure based on co-occurrence statistics of word forms. This is sometimes referred to in the literature as inferential meaning, as opposed to referential meaning. I prefer the term relational meaning, and see it in contrast to both adhesion and intention.

While I do not believe that relational meaning is fully equivalent to meaning, as the term is ordinarily used (and in academic discourse as well), I don’t wish to discuss that matter here. See my blog...
[Linkpost] Generalization in diffusion models arises from geometry-adaptive harmonic representation
by Bogdan Ionut Cirstea — 2023-10-11T17:48:24.500Z

This is a linkpost for https://arxiv.org/abs/2310.02557.

we show that two denoisers trained on non-overlapping training sets converge to essentially the same denoising function. As a result, when used for image generation, these networks produce nearly identical samples. These results provide stronger and more direct evidence of generalization than standard comparisons of average performance on train and test sets. The fact that this generalization is achieved with a small train set relative to the network capacity and the image size implies that the network’s inductive biases are well-matched to the underlying distribution of photographic images.

Here, we showed empirically that diffusion models can achieve a strong form of generalization, converging to a unique density model that is independent of the specific training samples, with an amount of training data that is small relative to the size of the parameter or input spaces. The convergence exhibits a phase transition between memorization and generalization as training data grows. The amount of data needed to cross this phase transition depends on both the image complexity and the neural network capacity (Yoon et al., 2023), and it is of interest to extend both the theory and the empirical studies to account for these. The framework we introduced to assess memorization versus generalization may be applied to any generative model.
What I've been reading, October 2023: The stirrup in Europe, 19th-century art deco, and more
by jasoncrawford — 2023-10-11T16:11:46.423Z
Link: https://rootsofprogress.org/reading-2023-10

A ~monthly feature. Last month was busy for me with a lot of travel and a lot of focus on The Roots of Progress as a nonprofit organization, so I haven’t had as much time as I prefer for research and writing. Recent blog posts and news stories are generally omitted; you can find them in my links digests. All emphasis in bold in the quotes below was added by me.

Histories of technology

Finished Joel Mokyr, The Lever of Riches: Technological Creativity and Economic Progress (1990), which I mentioned last time. The first part of the book is a summary of Western technological development from ancient times through the Industrial Revolution. The second part explores the causes of that development by looking at three contrasts: classical antiquity vs. the medieval period, Europe vs. China, and Britain vs. the rest of Europe.

The book is worth a full review, for now I’ll just leave you with one insightful quote, in the chapter where Mokyr considers the analogy between technological development and biological evolution:

The study of genetics is the study of the causes of genetic variation in the population. Yet genetics has contributed little to our understanding of speciation and nothing to our understanding of extinction (Lewontin, 1974, p. 12). Economic analysis, which postulates that techniques will be chosen by profit-maximizing firms employing engineers in whose minds the genotypes of various techniques are lodged, plays a role analogous to genetics. It explains how demand and supply produce a variety of techniques, and points to the constraining influences of environment and competition as a limit to the degree of variety. Just as genetics by itself does not explain speciation, economic analysis has difficulty explaining macroinventions. Like evolution, technological progress was neither destiny nor fluke. Yet the power of Darwinian logic—natural selection imposed on blind variation—is that we need not choose between the two.

I’m now about halfway through Lynn White, Medieval Technology and Social Change (1962)—another classic. White covers three major developments: the stirrup, the use of horses as draft animals, and the development of mechanical power. The focus is on the social change that these new technologies precipitated.

Famously, White argues that the stirrup created feudalism. The stirrup allowed the rider to brace himself more firmly on his horse, which ...
Attributing to interactions with GCPD and GWPD
by jenny — 2023-10-11T15:06:16.806Z

This post provides background, motivation, and a nontechnical summary of the purely mathematical https://arxiv.org/abs/2310.06686.

Coauthors (alphabetical): Chris MacLeod, Jenny Nitishinskaya, Buck Shlegeris. Work done mostly while at Redwood Research. Thanks to Joe Benton and Ryan Greenblatt for some math done previously. Thanks to Neel Nanda, Fabien Roger, Nix Goldowsky-Dill, and Jacob Hilton for feedback on various parts of this work.

Intro

In interpretability (and more generally in model understanding or model neuroscience) people care about measuring the effect on the model’s behavior from multiple inputs or components[1] (such as heads) and identifying which ones are important. This is called attribution.

Suppose we’ve done attribution to two different parts of the model. Intuitively, something very different is going on if these two parts are also importantly interacting than if they aren’t! In this post we consider the question: what is a principled interpretability framework for attributing to the interaction between inputs or components?

Summary

We can decompose a function into a sum of all the input interaction terms of various orders: the mean of the function, plus the individual contributions of each input, plus the second-order interaction of every pair of inputs, etc. This is the Generalized [Cumulant/Wick] Product Decomposition (G[C/W]PD).
Attribution to one input at a time is, in general, not enough to explain a function’s behavior.
If you aren’t measuring interactions, notice that you are assuming they are 0!
A potentially promising future direction is using this framework for mechanistic anomaly detection.

Background: attribution via interventions

Recall that we have a way to do attribution to model inputs (or components): tweak 1 part of the input while keeping the others the same. For example, to see how much a token in the input mattered, we can ablate that token and see how the model’s output changes.

In this post we are going to be talking in terms of resample ablation and taking expectations of the output over some distribution of your choice: for more info on why, see the causal scrubbing writeup.

FAQ

What’s the relevance for alignment?

NB this section is brief and elides a lot of detail, but this post felt incomplete without mentioning why the authors were interested in this direction.

We expect that when aligning superhuman systems, we may n...
You’re Measuring Model Complexity Wrong
by Jesse Hoogland — 2023-10-11T11:46:12.466Z

TLDR: We explain why you should care about model complexity, why the local learning coefficient is arguably the correct measure of model complexity, and how to estimate its value.

In particular, we review a new set of estimation techniques introduced by Lau et al. (2023). These techniques are foundational to the Developmental Interpretability research agenda and constitute the first generation of methods for detecting and understanding phase transitions, with potential applications for both interpretability and mechanistic anomaly detection. We expect this set of techniques to become a fixture in the alignment toolkit, and we've published a library and examples to help you get started.

This post is based on the paper, "Quantifying degeneracy in singular models via the learning coefficient" by Edmund Lau, Daniel Murfet, and Susan Wei (2023). The content builds on previous posts by @Liam Carroll on effective dimensionality in neural networks and the resulting perspective on phase transitions.

Why Model Complexity?

Model Comparison

Comparing models matters for safety. Given two models with the same behavior on a particular set of evals, we would like to be able to predict how they'll behave on out-of-distribution data. Can we distinguish the deceptively aligned model from the non-deceptively aligned model? As a first pass, can we predict that two models will or will not behave similarly in the future?

Comparing via weights. Because models are singular (parameters do not correspond one-to-one with functions), it's not possible to compare weights directly. Very different choices of weights can implement the same functions, and very similar choices of weights can implement qualitatively different functions.

Comparing via behavior. We also can't compare models at the level of the loss because different functions are compatible with the same loss. The same is true even if we compare models sample-by-sample: we'd need an astronomical number of inputs to meaningfully constrain behavior (some of which we'd rather not expose our model to).

Comparing via invariants. Another option is to take a pointer from the mathematicians and physicists: in classifying mathematical objects like spaces, it helps to turn geometric problems into algebraic problems where we can compute invariants. If two spaces have different values of that invariant, then we know that they're d...
Update on the UK AI Taskforce & upcoming AI Safety Summit
by Elliot Mckernon — 2023-10-11T11:37:42.436Z

While preparing for an upcoming Convergence Analysis post on the UK AI Taskforce and our recommendations, I looked into their reports, their plans for the upcoming AI safety summit, the recommendations from other orgs, and some miscellaneous UK AI events. I doubt we’ll include this in our post but I thought the updates were

worth sharing

, so here is a brief summary.

The UK AI taskforce

In April 2023, the UK government

committed £100 million

to its new

AI Foundation Models Taskforce

, led by Ian Hogarth. The taskforce was created in response to a white paper titled

A pro-innovation approach to AI regulation

in March, and was modeled on the

2020 Vaccine Taskforce

, with similar “agility and delegated authority”.

The government announced an

AI Safety Summit

on the 1st and 2nd of November at Bletchley Park, and put out a call for

expressions of interest

, looking for:

people with diverse skills and backgrounds to work in or with the Taskforce, to catalytically advance AI safety this year with a global impact. We're particularly interested in building out "safety infrastructure" and developing risk assessments that can inform policymakers and spur global coordination on AI safety.

In September, the taskforce released their

first report

. In summary:

Their expert advisory board consists of Yoshua Bengio, Paul Christiano, Matt Collins, Anne Keast-Butler, Alex van Someren, Helen Stokes-Lampard, and Matt Clifford.
Initially, their lone but influential AI researcher was

Nitarshan Rajkumar

, but now they're hiring technical AI experts including

Yarin Gal

, head of Oxford Applied & Theoretical ML Group, and

David Krueger

, leader of a Cambridge Deep Learning & AI Alignment group. They plan to scale up their team by an order of magnitude and encourage

applications

.
The taskforce is partnering with several organizations:

ARC Evals

, a non-profit focused on AI evaluations. The report emphasizes a close collaboration with ARC Evals in the leadup to the

summit

, along with

Redwood Research

and

RAND

.

Trail of Bits

, a cybersecurity consultancy. Their collaboration will be led by Heidy Khlaaf and will focus on cybersecurity around frontier AI systems.

The Collective Intelligence Project

, a non-profit governance tech incubator.

Center for AI Safety

, an AI safety non-profit focused o

...
An explanation for every token: using an LLM to sample another LLM
by Max H — 2023-10-11T00:53:55.249Z

Introduction

Much has been written about the implications and potential safety benefits of building an AGI based on one or more Large Language Models (LLMs). Proponents argue that because LLMs operate on human-readable natural language, they will be easier to understand, control, and scale safely compared to other possible approaches (e.g. MuZero-style model-based reinforcement learning and planning).

For example, suitably prompted, an LLM can output text which has the structure of an explanation followed by an answer or conclusion. Such methods have been shown to improve the accuracy of the final answer. They also give the consumer of the LLM's output the (apparent) ability to understand the LLM's reasoning process, which could boost trust and confidence relative to a bare answer.

One issue with this approach is that the natural language outputs of LLMs are not necessarily faithful representations of the underlying mechanisms which generated them. That is, the answer from an LLM may not depend on the preceding "explanation" or purported reasoning process at all.

Unfaithful explanations are actually an example of a more general issue with language models: in a long sequence of tokens, the later tokens in the sequence are often not constrained by earlier tokens as much as we would like. Additionally, the later tokens do not always depend on the earlier tokens in the natural or expected way, and in particular, do not depend on earlier outputs in the way the language model output itself claims.[1]

Some or all of these surface-level issues may be overcome by scaling the model and improvements in the model training process, but their underlying cause may be inherent to the way that all LLMs are currently used: auto-regressive sampling. The sampling process is a key part of LLM-based AI systems, and one that I think is currently somewhat overlooked as a useful place to impose powerful safety and control mechanisms.

In this post, I propose a new method for sampling an LLM by delegating the sampling process to a second LLM, and asking it to explain each choice. The result is that each token in the final output is accompanied by a human-readable out-of-band explanation for why that token was chosen, and thus the final output is more likely to be faithfully and causally related to purported explanations, compared to traditional Chain-of-Thought prompting and other in-band l...
Examples of Low Status Fun
by niplav — 2023-10-10T23:19:26.212Z

What is fun and what is high status are probably negatively correlated, and people probably mostly try to have high status fun. So I'd expect there to be several ways to have fun that are under-explored.

So, what are some example of things that are low status but potentially fun?

This is very much babble, but try to keep it to legal and mostly moral suggestions (though with less strong constraints on the latter).
A New Model for Compute Center Verification
by Damin Curtis — 2023-10-10T19:22:43.597Z

This post discusses a promising model for verifying compliance with international regulations on AI development. It is written by Damin Curtis & Alexander M. Wyckoff, and discusses a proposal by Yonadav Shavit.

The Verification Problem

To cope with emerging security challenges, states will have to create new regulatory frameworks to reign in the development of dangerous AI models/capabilities. To be effective, any laws or agreements will need a credible verification mechanism behind them, yet how to create such a mechanism is an open technical/policy question.

Frameworks to limit the proliferation of powerful weapons systems have been developed before, such as the International Atomic Energy Agency (IAEA) and the Nuclear Non-Proliferation Treaty (NPT). Through internationally agreed upon frameworks, inspections, and tracking of hazardous materials, the IAEA and NPT have successfully limited the development of nuclear WMD while allowing parties to develop useful technologies such as nuclear energy and biochemical research labs.

Developing a similar framework for verifying the peaceful development of AI models is, of course, difficult. The infrastructure and process of training safe versus unsafe models can be nearly identical, and unsafe protocols difficult to identify in training modules. What’s more, a method of verification must not endanger the privacy/intellectual property rights of the proving party (“the prover”). However, there is growing consensus that compute governance may be the keystone of AI governance, and there are strategic bottlenecks in the supply chain of compute-providing semiconductors that may allow for effective monitoring.

Proposal Overview

A recent (2023) proposal for such a framework by Yonadav Shavit seems promising on all of these fronts, as outlined in his paper,

What Does it Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training Via Compute Monitoring

. (Note that the following proposal is not our own, it is the work of Yonadav Shavit. The wording in this paper, and the discussion surrounding it, is our own.)

Shavit proposes that all advanced chips be manufactured with hardware which saves occasional “snapshots” of the training transcripts running on the chip, which can then be viewed by inspectors. An international regulatory agency would inspect to ensure chips are not ...
Announcing MIRI’s new CEO and leadership team
by Gretta Duleba — 2023-10-10T19:22:11.821Z

In 2023, MIRI has shifted focus in the direction of broad public communication—see, for example, our recent TED talk, our piece in TIME magazine “Pausing AI Developments Isn’t Enough. We Need to Shut it All Down”, and our appearances on various podcasts. While we’re continuing to support various technical research programs at MIRI, this is no longer our top priority, at least for the foreseeable future.

Coinciding with this shift in focus, there have also been many organizational changes at MIRI over the last several months, and we are somewhat overdue to announce them in public. The big changes are as follows.

Malo Bourgon: Chief Executive Officer (CEO)

Malo Bourgon, MIRI’s most long-standing team member next to Eliezer Yudkowsky, has transitioned from Chief Operating Officer into the senior leadership role at MIRI.[1] We piloted the change starting in February and made it official in June.
This is partly an attempt to better reflect long-standing realities at MIRI. Nate’s focus for many years has been on high-level strategy and research, while Malo has handled much of the day-to-day running of the organization.
This change also reflects that Malo is taking on a lot more decision-making authority and greater responsibility for steering the organization.

Nate Soares: President

Nate, who previously held the senior leadership role at MIRI (with the title of Executive Director), has transitioned to the new role of President.
As President (and as a board member), Nate will continue to play a central role in guiding MIRI and setting our vision and strategy.

Eliezer Yudkowsky: Chair of the Board

Eliezer, a co-founder of MIRI and a Senior Research Fellow, was already a member of MIRI’s board.
We’ve now made Eliezer the board’s chair in order to better reflect the de facto reality that his views get a large weight in MIRI strategic direction.
Edwin Evans, who was the board’s previous chair, remains on MIRI’s board.
Eliezer, Nate, and Malo have different senses of which technical research directions are most promising. To balance their different views, the board currently gives each of Eliezer, Nate, and Malo a budget to fund different technical research, in addition to the research that’s funded by the organization as a whole.

Alex Vermeer: Chief Operating Officer (COO)

Alex has stepped up to replace Malo as COO.
As COO, Alex is responsible for running/overse

...