trentmkelly/LessWrong-43k huggingface dataset:
Why do we need an understanding of the real world to predict the next tokens in a body of text?
by Valentin Baltadzhiev — 2024-02-06T14:43:50.559Z

I want to start by saying that this is my first question on LessWrong, so I apologise if I am breaking some norms or not asking it properly.

The whole question is pretty much contained in the title. I see a lot of people, Zvi included that claim we have moved beyond the idea that LLMs "simply predict the next token" and that they have some understanding now.

How is this obvious? What is the relevant literature?

Is it possible that LLMs are just mathematical representations of language? (does this question even make sense?) For example, if I teach you to count sheep by adding the number of sheep in two herds, you will get a lot of rules in the form of X + Y = Z, and never see any information about sheep. If, after seeing a million examples you become pretty good at predicting the next token in the sequence "5 + 4 =", does this imply that you have learned something about sheep?
On the Debate Between Jezos and Leahy
by Zvi — 2024-02-06T14:40:05.487Z

Previously: Based Beff Jezos and the Accelerationists

Based Beff Jezos, the founder of effective accelerationism, delivered on his previous pledge, and did indeed debate what is to be done to navigate into the future with a highly Worthy Opponent in Connor Leahy.

The moderator almost entirely stayed out of it, and intervened well when he did, so this was a highly fair arena. It’s Jezos versus Leahy. Let’s get ready to rumble!

I wanted to be sure I got the arguments right and fully stated my responses and refutations, so I took extensive notes including timestamps. On theme for this debate, this is a situation where you either do that while listening, or once you have already listened you are in practice never going to go back.

That does not mean you have to read all those notes and arguments. It is certainly an option, I found it interesting and worthwhile to study everything, if only sometimes on the level of an anthropologist, and to be sure I had gone the extra mile and had not missed anything.

There is however another option. Before I give my detailed notes, I will attempt a summary of the important takeaways from the debate, and attempt to build a model of what Jezos and Leahy were claiming and advocating. You can then check the transcript and notes for more details as desired.

Or you can Read the Whole Thing. If you do, I recommend skipping over the summary until after you have read the details, to see if your overall impressions match my own.

Actually Based Beff Jezos (ABBJ)

We were introduced in this debate to a character one could call Actually Based Beff Jezos (Analytical? Academic? Antihero? Apprehensive? Aligned?), or the Good Jezos, or the Motte Jezos, or the Reasonable Jezos.

Sometimes in this debate he is the one talking. Sometimes he is not.

I like Actually Based Jeff Jezos. We still very much have some issues, I think he is still importantly wrong about some very central things, but this would be someone I could be happy to work with or seek truth alongside in various ways. Actually Based Beff Jezos talks price.

Actually Based Beff Jezos starts from a bunch of positions that he takes farther than I would, but where I am much closer to his position than I am to the mainstream position on these questions:

He is a softspoken physicist and libertarian.

He has a deeply justified skepticism of government and institutions.

He

...
Why Two Valid Answers Approach is not Enough for Sleeping Beauty
by Ape in the coat — 2024-02-06T14:21:58.912Z

This is the sixth post in my series on Anthropics. The previous one is Another Non-Anthropic Paradox: The Unsurprising Rareness of Rare Events. The next one is Lessons from Failed Attempts to Model Sleeping Beauty Problem

Introduction

When I was writing about Anthropical Motte-Bailey, I had a faint hope that insights from it would be enough to solve the Sleeping Beauty paradox. But of course, it couldn't be that easy. My idea to simply raise the sanity waterline about anthropics to the point where providing a direct answer to this particular problem would be unnecessary also turned out to be wishful thinking. The discourse went for so long and accumulated so much confusion in process, that it requires much more direct attention.

So, now when I've written a couple of preliminary posts and hopefully persuaded you that anthropics problems are not special and can be solved in the realms of probability theory as long as we are being precise, let's do exactly that for the Sleeping Beauty problem.

But at first, I'll justify why the demand for an answer isn't unreasonable, and why we shouldn't be satisfied with the Two Valid Answers Approach. I'll show what issues this approach has, and how it can be used to highlight the crux of disagreement between two positions. And, given all that, what it would actually mean to solve the Sleeping Beauty problem.

Two Valid Answer Approach

According to the Two Valid Answers Approach, both 1/2 and 1/3 are correct answers, but to different questions. The problem is ambiguously formulated: both of these questions can be considered valid interpretations, and both answers are relevant to different decision theory problems.

The situation is not unlike the infamous fallen tree in the forest question. We just need to talk about specific betting odds and stop trying to answer the general and ambiguous question about probability.

I am sympathetic towards this perspective. In my first post on anthropics, I specifically dedicated some place to explain how different scoring rules can produce different probabilities in the Sleeping Beauty setting.

But I think it is not the whole story. Two Valid Answers Approach resolves some of the confusion - yes, but it also attempts to hide the rest of it under the rug.

The issue is that people read such explanations, nod in agreement and then do not really change their initial position...
Are most personality disorders really trust disorders?
by chaosmage — 2024-02-06T12:37:56.070Z

Unlike other social species, we trust selectively; we choose who we cooperate with. We have exerted evolutionary pressure on each other to to judge well who is worthy of our trust, as well as evolutionary pressure to be (selectively) trustworthy. For more detail, see The biological function of love for non-kin is to gain the trust of people we cannot deceive.

This is not true for our closest primate relatives, so it is an evolutionarily recent phenomenon. We can therefore expect it is not mature and stable across the entire population. I believe this is indeed observable, and propose that most personality disorders are how we perceive people in whom this ability is malfunctioning.

Perso-nality
disor-der	Phenotype (summary)	Malfunction of trust cognition
Anti-social	A long-term pattern of manipulating, exploiting, or violating the rights of others	Inability or near-inability to care and therefore be trustworthy.
Avoi-dant	Very shy; feels inferior a lot; usually avoids people due to fear of rejection	Too low expectation of trust from others.
Bor-der-line	Lots of trouble managing emotions; impulsive; uncertain self-image; very troubled relationships.	Randomness in computation of trust and trustworthiness.
De-pen-dent	Over-dependence on others; may let others treat them badly out of fear of losing the relationship.	Too high expectation others will care and therefore be trustworthy.
Hist-rionic	Dramatic, strong emotions, always wanting attention from others.	Too low expectation others will care and therefore be trustworthy.
Nar-cissis-tic	Lacks empathy, wants to be admired by others, expects special treatment.	Too high expectation of trust from others.
Para-noid	Extreme fear and distrust of others.	Inability or near-inability to trust.
Schiz-oid	Prefers to be alone; disinterest in having relationships with others.	Understanding trust is too effortful, so often not worth the trouble.

However, depressive, obsessive-compulsive and schizotypal personality disorder do not fit neatly into this frame. I propose these are different things, namely persistent forms of depression, OCD and schizophrenia respectively, that are mild enough not to have been weeded out of the gene pool despite their persistence.

So why consider this hypothesis of the pathogenesis of these disorders? Because it takes eight problems currently considered tied up with personal identy and essentially unsolvable (although talk ther...
From Conceptual Spaces to Quantum Concepts: Formalising and Learning Structured Conceptual Models
by Roman Leventov — 2024-02-06T10:18:40.420Z
Link: https://arxiv.org/abs/2401.08585

Submitted on 6 Nov 2023

Sean Tull, Razin A. Shaikh, Sara Sabrina Zemljic, Stephen Clark

Abstract

In this article we present a new modelling framework for structured concepts using a category-theoretic generalisation of conceptual spaces, and show how the conceptual representations can be learned automatically from data, using two very different instantiations: one classical and one quantum. A contribution of the work is a thorough category-theoretic formalisation of our framework. We claim that the use of category theory, and in particular the use of string diagrams to describe quantum processes, helps elucidate some of the most important features of our approach. We build upon Gardenfors' classical framework of conceptual spaces, in which cognition is modelled geometrically through the use of convex spaces, which in turn factorise in terms of simpler spaces called domains. We show how concepts from the domains of shape, colour, size and position can be learned from images of simple shapes, where concepts are represented as Gaussians in the classical implementation, and quantum effects in the quantum one. In the classical case we develop a new model which is inspired by the Beta-VAE model of concepts, but is designed to be more closely connected with language, so that the names of concepts form part of the graphical model. In the quantum case, concepts are learned by a hybrid classical-quantum network trained to perform concept classification, where the classical image processing is carried out by a convolutional neural network and the quantum representations are produced by a parameterised quantum circuit. Finally, we consider the question of whether our quantum models of concepts can be considered conceptual spaces in the Gardenfors sense.

Parts of the introduction:

The study of concepts has a long history in a number of related fields, including philosophy, linguistics, psychology and cognitive science (Murphy, 2002; Margolis & Laurence, 2015). More recently, researchers have begun to consider how mathematical tools from quantum theory can be used to model cognitive phenomena, including conceptual structure. The general use of quantum formalism in psychology and cognitive science has led to an emerging area called quantum cognition (Aerts, 2009; Pothos & Busemeyer, 2013). The idea is that some of the features of quantum theory, such as entanglement, ca

...
Fluent dreaming for language models (AI interpretability method)
by tbenthompson — 2024-02-06T06:02:59.296Z
Link: https://arxiv.org/pdf/2402.01702.pdf

This is a cross-post for our paper on fluent dreaming for language models. (arXiv link.) Dreaming, aka "feature visualization," is a interpretability approach popularized by DeepDream that involves optimizing the input of a neural network to maximize an internal feature like a neuron's activation. We adapt dreaming to language models.

Past dreaming work almost exclusively works with vision models because the inputs are continuous and easily optimized. Language model inputs are discrete and hard to optimize. To solve this issue, we adapted techniques from the adversarial attacks literature (GCG, Zou et al 2023). Our algorithm, Evolutionary Prompt Optimization (EPO), optimizes over a Pareto frontier of activation and fluency:

In the paper, we compare dreaming with max-activating dataset examples, demonstrating that dreaming achieves higher activations and similar perplexities to the training set. Dreaming is especially exciting because some mildly out-of-distribution prompts can reveal details of a circuit. For example, Pythia-12B layer 10, neuron 5 responds very strongly to "f. example", "f.ex." and "i.e" but responds even more strongly to "example f.ie.", a phrase the model has probably never seen in training.

Figure: Comparing activation and cross-entropy between dreaming outputs and the top 64 max-activating dataset examples from 500 million tokens of the Pile. Lower cross-entropy prompts are more fluent. The black line is schematically separating regions of the plot that are empirically inside and outside the training distribution.

Like max-activating dataset examples, language model dreams will be hard to interpret in the face of polysemanticity. We would be excited about applying dreaming to more monosemantic feature sets resulting from dictionary learning/sparse autoencoders.

We think algorithms like EPO will also be useful for fluent algorithmic redteaming. We are working on that now!

We have a companion page here demonstrating the code. We also have a demo Colab notebook here.
Toy models of AI control for concentrated catastrophe prevention
by Fabien Roger — 2024-02-06T01:38:19.865Z

In this post, we analyze some simple games that model the problem of ensuring AIs never succeed at taking a single catastrophic action (which we'll refer to as "escaping") during deployment. This exercise highlights some of the difficulties associated with deploying misaligned models given a limited perfect auditing budget, and also suggests some opportunities for technique development.

We analyzed these games while writing AI Control: Improving Safety Despite Intentional Subversion. (Thanks to harfe for solving one of these games, and to Ryan Greenblatt and Kshitij Sachan for discussion about these at the time.)

Core takeaways:

Indistinguishable episodes (i.e. the AI doesn't know what time it is) + tiny escape success rate + catch-then-shut-down can get high chances of win even with a small fraction of auditing, but only if the success rate is really tiny or if failures are loud (i.e. we notice failed attempts).
We can improve our chances by identifying on which episodes it would be harder to escape and auditing them more often. This works even if the AI knows which episodes we think are good opportunities.
We can improve our chances significantly if a escape requires collusion, as long as you audit at least a few examples from the category of data on which AIs plan to escape.

Assumptions used across all toy models

For the sake of simplicity, we make the following assumptions:

The situation is a zero-sum game between a scheming AI and humanity, where the scheming AI wins if it takes over, and humanity wins if it either catches the scheming AI red-handed, or if the deployment period ends without the scheming AI having successfully taken over.
We analyze these scenarios as games where the humans first choose a strategy (which can be randomized) and then the AI chooses a response. This means that we’re assuming that both players are choosing the optimal strategy, and also assuming that the AI knows what game it’s playing and all the parameters of the game (e.g. the human auditing budget, the number of episodes in the game, etc).
The only failures we model are concentrated catastrophic failures, which take a single catastrophic action to enable the deceptive AI to escape.
The AI doesn’t maintain any state, and so e.g. can’t learn from evaluations (but can learn from data on which we trained it).

Distinguishable episodes with limited auditing

The model: In every episod...
Things You’re Allowed to Do: University Edition
by Saul Munn — 2024-02-06T00:36:11.690Z
Link: https://www.brasstacks.blog/tyatd-uni/

This post is not titled “Things You Should Do,” because these aren’t (necessarily) things you should do. Many people should not do many of the items on this list, and some of the items are exclusive, contradictory, or

downright the reverse of what you should do

. If your reaction to something is “I think that’s a bad idea,” then it probably is, and you probably shouldn’t do it.

classes & professors

attend classes you haven’t signed up for because you find them interesting
attend classes even if the waitlist is full
ask the professor to waive a prerequisite
ask the professor to join a class even if its full
drop a class that you don’t like
take a class because you really liked the professor, even if you’re not sure about the content of the class

cold email

professors you don’t know, just asking to chat
show up to office hours for classes you aren’t a part of, just to chat with the professor
ask the professor questions about the things you’re not sure of
skip class(es) for great opportunities elsewhere
ask the professor if you can help them with anything in the class (grading, setting up assignments, editing papers, etc). professors have a long list of tasks, are perpetually behind, and encounter fairly correlated problems; if you track what problems your professors have, you can quite quickly become unreasonably useful for them
ask professors at the beginning of the semester what things would be most important to memorize, then throw their answers into an

Anki deck

take non-credit courses or workshops in things like pottery, coding, or creative writing

studying

at places outside of your university:

coffeeshops
public libraries
coworking spaces
random offices,

cold email

them

start a study group for the class

ask the professor if you can announce that you’re starting a study group for the class in the class
start a group chat to ask questions about the class. this is one that everyone loves to be added to, and sometimes it just… doesn’t happen, because nobody took the initiative to create it

use

Anki

to study the things your professor said would be most important to memorize after you asked them at the beginning of the semester
learn the content of a class by using materials that the professor doesn’t point you toward (e.g. online textbooks/videos/tutors/etc)
hire a tutor

hire multiple tutors
hire a tutor purely so that you have to study for some class you hate — you might not ne

...
Value learning in the absence of ground truth
by Joel_Saarinen — 2024-02-05T18:56:02.260Z

Epistemic Status: My best guess.

Summary

The problem of aligning AI is often phrased as the problem of aligning AI with human values, where “values” refers roughly to the normative considerations one leans upon in making decisions. Understood in this way, solving the problem would amount to first articulating the full set of human values that we want AI systems to learn, and then going about actually embedding these into the systems. However, it does not seem to be the case that there is a single set of human values we’d want to encode in systems in the first place, or a single, unchanging human reward function that we want systems to learn and optimize for. The absence of a target for learning intuitively poses problems for learning human values and thus for solving alignment more generally. Nonetheless, as humans are still capable of acting in ways that we would want AI systems to act even in the absence of a “ground truth” to human values, and assuming that this capability is learned to at least some extent, alignment still seems possible, albeit through value-learning processes that reject this assumption of a ground truth (contrary to the current value-learning paradigm). I review four such proposed approaches, discuss them in the context of AI safety, and consider the long-term prospects of these kinds of approaches bringing about aligned AI.

Summaries of each framework are included at the beginning of each respective section. For those interested only in the conclusions, skip to the section “Taking stock”above.

Introduction

Value learning is one of the better-known approaches to tackling the alignment problem. In a nutshell, this involves getting AI systems to learn to value what their human operators want them to value from training data. One way of doing this is through inverse reinforcement learning (IRL), whereby after observing a human enacting the desired type of behavior in the environment, the AI system infers the reward function that the human model is trying to maximize. By either scaling the techniques generating these inferences or doing IRL on enough tasks, the hope is that AI systems will start to form an idea of a more general utility function for humans, and thus in executing any given task in the world, they will do it in a way that either humans themselves would or in a way they would approve of.

A classic example of violating t...
Implementing activation steering
by Annah — 2024-02-05T17:51:55.851Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Autumn 2023 Cohort and while being an affiliate at PIBBSS in 2024. A thank you to @Jayjay and @fela for helpful comments on this draft.

This blog post is an overview of different ways to implement activation steering with some of my takes on their pros and cons. See also this GitHub repository for my minimal implementations of the different approaches.

The blog post is aimed at people who are new to activation/representation steering/engineering/editing.

General approach

The idea is simple: we just add some vector to the internal model activations and thus influence the model output in a similar (but sometimes more effective way) to prompting.

Example[1]: Imagine that some vector in the internal representations in some transformer layer encodes a direction associated with "Love". When you add this vector to the activations of some encoded sentence "I hate the world", you change the internal representation (and thus the meaning) to something more like "I love the world". This graphic might help with an intuition:

In general there are a few steps involved which I simplify in the following:

Decide on a layer l and transformer module ϕ to apply the activation steering to.
Define a steering vector. In the simplest case we just take the difference of the activations of two encoded strings like v=ϕl(Love)−ϕl(Hate).
Add the vector to the activation during the forward pass. In the simplest case it's something like ~ϕl=ϕl+v.

Each of the three points mentioned above includes complexities you might encounter as a beginner. Feel free to move on to the next section if you prefer.

You can do activation steering at pretty much any layer/module in the transformer. It's often done at the residual stream of one of the hidden layers. However, if you want to do activation steering by modifying the bias parameter, you need to do it in a transformer module that has a specific structure. This is usually not the residual stream but one can do it in the attention or the mlp module.
When defining a direction there are several things that might complicate it:

Tokens: When encoding a word or a short sentence it is often encoded into several tokens, so when you get to the internal activation ϕl(Love) you don't just have one vector but one vector per token. Even worse, 'Love' a

...