trentmkelly/LessWrong-43k huggingface dataset:
Entanglement and intuition about words and meaning 
by Bill Benzon — 2023-10-04T14:16:29.713Z

Cross-posted from New Savanna.

Two things have just occurred to me about my recent post, Word meaning and entanglement in LLMs:

1.) That the issue is one of intuition as well, and
2.) that we’re dealing with system 1 thinking, in the System 1/system 2 dichotomy popularized by Daniel Kahneman.

I’ve not yet read Kahneman’s book – Thinking Fast, Thinking Slow, though it’s on my “to be read someday” shelf – but I gather that System 1 is fast, intuitive and largely tacit (to use a word from Michael Polanyi) while System two is slow, deliberate, and logical.

My argument in that earlier post is that, in effect, our default notion of word meaning is that it is atomic and discrete. When words are linked in phrases, sentences, and paragraphs, it is liking beads on a thread, or freight cars in a train. The linkage is external and contingent. Without reflection, that’s just how we think about words (and meaning). That’s fine in informal discussions, but not so good in at least some technical contexts, such as large language models (LLMs).

Now, let’s take the idea that LLMs are “trained” by being asked to predict the next word. That is at least consistent with, if not actually reinforcing of, this default conceptualization of atoms-of-meaning. One can easily make predictions about the behavior of atoms. One simply observes them and notes down what they do from one moment to the next. There is no sense of “interiority.”

Whereas the idea that words are entangled with one another through their meanings, that’s all about “interiority.” Those vectors are “interior” to the token, and relate one token to another and, more generally, tokens among themselves. The idea of entanglement leads naturally to the idea of weaving, weaving a fabric of meaning. The so-called prediction procedure, then, is one of placing a word, with its 12K item vector, into the unfolding fabric of meaning. Backpropagation, in this view, is the act of fine-tuning the placement. Prediction is merely a means to an end, a device, not the point of the procedure.

To think in terms of atomic meaning is simply to gloss over all this. All of that may be implicit in the mathematics, but the atomic view of meaning stands in the way of allowing ones thought to be perspicuously guided by the mathematics. The mathematics becomes (and functions as) a secondary construction.

I note finally that traditional training in proposition...
What are some examples of AIs instantiating the 'nearest unblocked strategy problem'?
by EJT — 2023-10-04T11:05:34.537Z

A paragraph explaining the problem, from Ngo, Chan, and Mindermann (2023). I've bolded the key part:

Our definition of internally-represented goals is consistent with policies learning multiple goals during training, including some aligned and some misaligned goals, which might interact in complex ways to determine their behavior in novel situations (analogous to humans facing conflicts between multiple psychological drives). With luck, AGIs which learn some misaligned goals will also learn aligned goals which prevent serious misbehavior even outside the RL fine-tuning distribution. However, the robustness of this hope is challenged by the nearest unblocked strategy problem [Yudkowsky, 2015]: the problem that an AI which strongly optimizes for a (misaligned) goal will exploit even small loopholes in (aligned) constraints, which may lead to arbitrarily bad outcomes [Zhuang and Hadfield-Menell, 2020]. For example, consider a policy which has learned both the goal of honesty and the goal of making as much money as possible, and is capable of generating and pursuing a wide range of novel strategies for making money. If there are even small deviations between the policy’s learned goal of honesty and our concept of honesty, those strategies will likely include some which are classified by the policy as honest while being dishonest by our standards. As we develop AGIs whose capabilities generalize to an increasingly wide range of situations, it will therefore become increasingly problematic to assume that their aligned goals are loophole-free.

LLMs being vulnerable to jailbreaks seems like a decent example. Are there others?
Graphical tensor notation for interpretability
by Jordan Taylor — 2023-10-04T08:04:33.341Z

[ This post is now on arXiv too: https://arxiv.org/abs/2402.01790 ]

Some examples of graphical tensor notation from the QUIMB python package

Deep learning consists almost entirely of operations on or between tensors, so easily understanding tensor operations is pretty important for interpretability work.[1] It's often easy to get confused about which operations are happening between tensors and lose sight of the overall structure, but graphical notation[2] makes it easier to parse things at a glance and see interesting equivalences.

The first half of this post introduces the notation and applies it to some decompositions (SVD, CP, Tucker, and tensor-network decompositions), while the second half applies it to A Mathematical Framework for Transformer Circuits. Most of the first half is adapted from my physics PhD thesis introduction, which is partly based on existing explanations such as in the math3ma blog, Simon Verret's blog, tensornetwork.org, tensors.net, Hand-waving and Interpretive Dance: An Introductory Course on Tensor Networks, An Intuitive Framework for Neural Learning Systems, or a talk I gave in 2021. Feel free to scroll around and look at interesting diagrams rather than reading this post start to finish.

Tensors

Practically, tensors in our context can just be treated as arrays of numbers.[3] In graphical notation (first introduced by Roger Penrose in 1971), tensors are represented as shapes with "legs" sticking out of them. A vector can be represented as a shape with one leg, a matrix can be represented as a shape with two legs, and so on. I'll also represent everything in PyTorch code for clarity.

Each leg corresponds to an index of the tensor - specifying an integer value for each leg of the tensor addresses a number inside of it:

where 0.157 happens to be the number in the (i=0,j=2,k=0) position of the tensor A. In python, this would be A[0,2,0]. The amount of memory required to store a tensor grows exponentially with the number of legs,[4] so tensors with lots of legs are usually represented only implicitly: decomposed as operations between many smaller tensors.

Operations

The notation only really becomes useful when things get more complicated, but let's start as simple as possible. Multiplying two numbers together (y = a * b) in graphical tensor notation just involves drawing them nearby:

The next easies...
[Link] Bay Area Winter Solstice 2023
by tcheasdfjkl — 2023-10-04T02:19:56.284Z
Link: https://fb.me/1QOVAoRFRHROxgK

Bay Area Winter Solstice 2023 will be held on December 16 at the Freight & Salvage in downtown Berkeley!

Tickets are available at https://www.eventbrite.com/e/bay-area-winter-solstice-2023-tickets-721678057497.

See the official event page at https://fb.me/1QOVAoRFRHROxgK for more information and future updates.
Who determines whether an alignment proposal is the definitive alignment solution?
by MiguelDev — 2023-10-03T22:39:23.700Z

What is the established process, or what could be a potential process, for validating a highly probable alignment solution? I assume that a robust alignment solution would undergo some form of review—so who is responsible for reviewing these proposals? Is LessWrong the platform for this? or is there a specialized communication network that researchers can access should such a situation arise?

For now, the option I see is that the burden falls on the researcher to demonstrate the solution in a real-world setting and have it peer reviewed. However, this approach risks exposing the techniques used, making them available for capabilities research as well.

If there is already a post addressing this question, please share it here. Thank you.
AXRP Episode 25 - Cooperative AI with Caspar Oesterheld
by DanielFilan — 2023-10-03T21:50:07.552Z

YouTube link

Imagine a world where there are many powerful AI systems, working at cross purposes. You could suppose that different governments use AIs to manage their militaries, or simply that many powerful AIs have their own wills. At any rate, it seems valuable for them to be able to cooperatively work together and minimize pointless conflict. How do we ensure that AIs behave this way - and what do we need to learn about how rational agents interact to make that more clear? In this episode, I’ll be speaking with Caspar Oesterheld about some of his research on this very topic.

Topics we discuss:

Cooperative AI

… vs standard game theory

Do we need cooperative AI if we get alignment?

Cooperative AI and agent foundations

A Theory of Bounded Inductive Rationality

Why it matters

How the theory works

Relationship to logical induction

How fast does it converge?

Non-myopic bounded rational inductive agents?

Relationship to game theory

Safe Pareto Improvements

What they try to solve

Alternative solutions

How safe Pareto improvements work

Will players fight over which safe Pareto improvement to adopt?

Relationship to program equilibrium

Do safe Pareto improvements break themselves?

Similarity-based Cooperation

Are similarity-based cooperators overly cliqueish?

Sensitivity to noise

Training neural nets to do similarity-based cooperation

FOCAL, Caspar’s research lab

How the papers all relate

Relationship to functional decision theory

Following Caspar’s research

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Caspar Oesterheld. Caspar is a PhD student at Carnegie Mellon where he’s studying with Vincent Conitzer. He’s also the assistant director of the Foundations of Cooperative AI Lab or FOCAL. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net. All right. So welcome to the podcast, Caspar.

Caspar Oesterheld:
Thanks. Happy to be on.

Cooperative AI

Daniel Filan:
Yeah. So my impression is that the thing tying together the various strands of research you’re involved in is something roughly along the lines of cooperative AI. Is that fair to say?

Caspar Oesterheld:
I think that’s fair to say. I do some ...
When to Get the Booster?
by jefftk — 2023-10-03T21:00:12.813Z

Let's say you're planning on getting a covid booster this fall: when's
the best time to get it? The updated boosters (

targeting
the XBB lineage) have been out since mid September, so there's not
a new vaccine to wait for. Instead, I see the choice as balancing two
considerations:

If you get it too soon it might have worn off too much by the
time you most need it.

But if you're too late you might get infected in the meantime.

As a first approximation, you probably want to have the strongest
protection when local levels be at their highest. When would that be?
Wastewater monitoring is pretty good for this sort of thing because
it's not dependent on people getting tested. Here's what I see on Biobot:

It looks like 2020-2021 and 2021-2022 were strongly concentrated
around New Years, and 2022-2023 less so. On the other hand,
2023-2024 so far is following a trend very close to 2021-2022, so
perhaps it will be up for the holidays again?

The other key question here is how quickly the vaccine wears off. It
looks like the most recent meta-analysis here is Menegale
et. al 2023, which found effectiveness decreased quite rapidly
against Omicron (and everything now is a kind of Omicron):

They estimated a half life of 111d [88-115d]. This means that if you
got a shot on the first day they were made available this year
(2023-09-12) you'd be down to 50% [42-51%] effectiveness at New
Years. I wish the CDC would be more transparent about their reasoning
so we could tell whether this was on purpose...

At this point I'd love to see a calculator that lets you put in when
you last got a booster (or had covid) and then combined the half life
data with the historical seasonality data to identify the
covid-minimizing time to get a shot. It could even allow you to
specify dates you want to not be sick for, or not get sick during,
along with how important it is to you.

Unfortunately this calculator doesn't exist, so we'll have to eyeball
it. I think most people would like to avoid infection around
Thanksgiving and Christmas, historically high-infectious times that we
especially don't want interrupted by covid and during which we're much
more likely than usual to be getting together in large
multigenerational groups. Getting a shot two weeks before
Thanksgiving, 2023-11-09, would have you at most protected for
Thanksgiving, and the...
OpenAI-Microsoft partnership
by Zach Stein-Perlman — 2023-10-03T20:01:44.795Z

OpenAI has a

strong

partnership

with Microsoft. The details are opaque, as far as I know. It tentatively seems that OpenAI is required to share its models (and some other IP) with Microsoft until OpenAI

attains

"a highly autonomous system that outperforms humans at most economically valuable work."[1] This is concerning because AI systems could cause a catastrophe with capabilities below that threshold.

(OpenAI may substantially depend on Microsoft; in particular, Microsoft Azure

is

"OpenAI's exclusive cloud provider." Microsoft's power over OpenAI may make it harder for OpenAI to refuse to share dangerous systems with Microsoft. But mostly this seems moot if OpenAI is just straightforwardly required to share its models with Microsoft.)

If so, then (given that Microsoft is worse on safety than OpenAI) whether OpenAI would do good alignment between training and deployment and then deploy cautiously mostly doesn't matter, because (if OpenAI is leading near the end) whether unsafe AI is deployed will be determined by Microsoft's decisions?

[Edit: I don't think Microsoft has full real-time access to OpenAI's models, given that they launched Bing Chat after OpenAI had RLHF'd GPT-4 but Bing Chat wasn't based on that version of GPT-4, as well as some other reporting. But it's very unclear what access Microsoft does have, or why OpenAI and Microsoft aren't transparent about this.]

(The OpenAI-Microsoft relationship seems like a big deal. Why haven't I heard more about this?)

Update: Microsoft and OpenAI have a a joint Deployment Safety Board (

https://blogs.microsoft.com/on-the-issues/2023/10/26/microsofts-ai-safety-policies/

,

https://openai.com/global-affairs/our-approach-to-frontier-risk

,

https://www.theverge.com/23733388/microsoft-kevin-scott-open-ai-chat-gpt-bing-github-word-excel-outlook-copilots-sydney

). The details are opaque.

^

OpenAI says:

by AGI we mean a highly autonomous system that outperforms humans at most economically valuable work. Such a system is excluded from IP licenses and other commercial terms with Microsoft, which only apply to pre-AGI technology.

It's not clear whether OpenAI has to share everything besides AGI with Microsoft.
Current AI safety techniques?
by Zach Stein-Perlman — 2023-10-03T19:30:54.481Z

What safety techniques exist? What techniques should labs be using, now or if they soon develop powerful models? What techniques should be considered industry-best-practices? "How might we align transformative AI if it’s developed very soon?" What's our current bag of tricks? What alignment research is ready to go?

Some general sources:

Buck's The current alignment plan, and how we might improve it (especially v1 Plan and Classes of technical research) (and Redwood's unpublished proposals on control)
Holden's How might we align transformative AI if it’s developed very soon? (especially Key tools)

Some sources on particular (kinds of) techniques:

RLHF
Activation engineering

Some categories of safety techniques:

Before-training safety techniques (e.g. architecture & dataset)
Before-deployment safety techniques

During training + between training and deployment

(Before-deployment threat detection)

(Also important is detecting dangerous properties of the model — the current kind-of-technique is model evals or red-teaming mostly done by humans; maybe someday there will be AI-based red-teaming or interpretability. This is noncentral in this post but I'm interested in specific techniques.)

During-deployment, inference-time control/monitoring

(Including not just techniques prompting and chain-of-thought and monitoring-outputs-for-coups, but also tools or stuff-you-build like training a classifier and building interpretable agent-scaffolding)

This all depends on threat models and how the leading lab would use its powerful AI to prevent others from training/deploying unaligned powerful AI (or make the world robust to unaligned powerful AI)...

Some notes or pointers-to-sources on what OpenAI and Anthropic say they do:

OpenAI: they use RLHF.[1] While working on

GPT-4

, they "used GPT-4 to help create training data for model fine-tuning and iterate on classifiers across training, evaluations, and monitoring"; see also Using GPT-4 for content moderation.

"Risks & mitigations" in "GPT-4 Technical Report"

discusses "Adversarial Testing via Domain Experts" and "Model-Assisted Safety Pipeline"; see also

"Model Mitigations" in "GPT-4 System Card."

OpenAI ultimately has "Superalignment" goals, but they say "Our plan in the shorter term is to

use AI to help humans evaluate

the outputs of more complex models and monitor complex systems."[2][3]

Anthropic: they use

Cons

...
Metaculus Announces Forecasting Tournament to Evaluate Focused Research Organizations, in Partnership With the Federation of American Scientists 
by ChristianWilliams — 2023-10-03T16:44:17.620Z
Link: https://www.metaculus.com/questions/19150/fro-casting-to-support-a-new-research-model/

You are invited to participate in Metaculus's FRO-Casting Tournament, an exciting pilot project in partnership with The Federation of American Scientists that harnesses predictions to help assess impact, deliver feedback, and inform the allocation of $50 million to ambitious research proposals handpicked by Convergent Research.

There is a wealth of untapped scientific knowledge at the intersection of research and engineering: Industry doesn't pursue it because it's unlikely to be profitable. Academia doesn't pursue it because it's unlikely to be publishable. Enter Focused Research Organizations (FROs), non-profits modeled after startups that address well-defined technology challenges that are unlikely to be solved by industry or academia.

By sharing your insights, you'll join forces with Metaculus Pro Forecasters and 25 subject matter experts to help generate:

Risk-reward profiles of each FRO proposal
Actionable feedback for proposal authors
Valuable data on forecasting's role in the scientific review process

Learn more about the specific proposals below. We recommend reading each proposal's attached forecasting guide before getting started.

Or, jump right into forecasting here.

Antibody Characterization through Open Science

"Many antibodies that scientists purchase from commercial manufacturers to conduct their research do not work as advertised, because most have never been validated properly. This project brings together the public and private sectors to conduct independent, third-party testing of commercial antibody manufacturers' catalogs and publish the results in the public domain, such that no scientist ever uses an ineffective antibody again." Read the full research proposal.

Proposal Forecasting Guide

Get Started

Modular and Scalable Platform for Human Molecular Monitoring

"Wearable health electronics are now ubiquitous, but continuous molecular monitoring is only widely available for glucose. Decades of research have expanded continuous monitoring to other molecules, but these techniques are restricted to research labs and remain disconnected from daily human use. We propose a platform to translate and distribute these emerging technologies, enabling the mapping of the time-varying human metabolome and the design of closed-loop devices for personalized health." Read the full research proposal.

Proposal Forecasting Guide

Get Started

R

...
What would it mean to understand how a large language model (LLM) works? Some quick notes. 
by Bill Benzon — 2023-10-03T15:11:13.508Z

Cross-posted from New Savanna.

I don’t mean “understand” in any deep philosophical sense. I mean only a rough and ready sense of the word. We understand how toasters work, automobiles, moon rockets, digital computers, and so forth. We know how to design and construct these things, how to diagnose problems, how to maintain and repair them. Not perfectly to be sure, but well enough to use these devices to get things done.

LLMs, however, are said to be opaque. We don’t know how they work. We feed them prompts, they produce output, but how the model works from the prompt to produce the output, that’s mysterious. There are people working on mechanical interpretability, trying to understand the LLM as though it were a machine, or at least, a computer program of the ordinary kind, where we know, more or less, how it works on data – if it is the kind of program that works from data – to produce output. But what would it mean to understand the operational characteristics of 175 billion parameters, as in the case of GPT-3.5?

It means, I suppose, how those parameters mediate between the input, a prompt, and the output, whatever “follows from” a given prompt. At the lowest level we are told that LLMs are prediction machines. So, the output string is simply a continuation of the input string. And I suppose that, technically, that’s true. But it’s not very helpful, as I’ve argued at some length.

Let’s set that aside.

What could we possibly want by way of understanding?

We’ve got three things: There is the underlying engine, let’s call it, which is a computer program like any other. It’s created by programmers working with some language or languages and is designed to achieve a certain purpose. In this case, it’s designed to create a language model over a corpus of texts and then to use that model in generating new chunks of language given an input prompt.

It's that model that’s problematic, that’s said to be opaque. We, us humans, didn’t create that model. The engine did. And, in the case of GPT-3, that model’s got 175 billion parameters. More recent models have even more. And there are also models with only millions of parameters. But even those smaller models are huge.

But, here’s the thing, how can we understand how that opaque model operates unless we understanding what it’s trying to do? Sure, we can pop the hood and take a look. We see a bunch of gizmos, widgets, framblasts...
Potential alignment targets for a sovereign superintelligent AI
by Paul Colognese — 2023-10-03T15:09:59.529Z

I'd like to compile a list of potential alignment targets for a sovereign superintelligent AI.

By an alignment target, I mean something like what goals/values/utility function we might want to instill in a sovereign superintelligent AI (assuming we've solved the alignment problem).

Here are some alignment targets I've come across:

Alignment to a human user (or group of human users).
Ambitious value learning.
Coherent extrapolated volition.

Examples, reviews, critiques, and comparisons of alignment targets are welcome.