trentmkelly/LessWrong-43k huggingface dataset:
Cartography, blowing one's mind, the illusion of separation and other general musings
by Neil  — 2023-06-16T19:19:00.725Z

Or: LessWrong cartography, the illusion of separation, and blowing one's mind

TLDR: At LessWrong, we make maps. To make maps, we carve the universe out into shards: this is your daily reminder that you can (mostly) carve the shards out in whatever way you want.

Disclaimer: I came up with these ideas for fun. I knew there was something useful within them, but I had to go through several drafts of this post to understand what my point was. I hope the final result isn't too bad, just keep in mind this is a kind-of messy soup of ideas.

1: LessWrong is about making reasonably accurate maps of reality for individuals to use. In order to make them, you have to divide the universe into shards and then assemble them together into a lattice that will faithfully guide you through reality.

2: Blowing your mind is a useful kind of fun. It means finding something so much bigger than you that discovering it and then understanding it for a short while fills you with unutterable awe.

3: The best shortcut to blowing your mind is breaking down separations: realizing that separations don't fundamentally exist, and then basking in the strange kind of chaotic freedom this gives you. This Feynman quote puts it well:

If our small minds, for some convenience, divide this universe, into parts — physics, biology, geology, astronomy, psychology, and so on — remember that nature does not know it!

4: Our brains have evolved and are trained to split the white noise of the universe into shards. There may be a million "pixels" in your field of vision right now but your brain probably picks up only a dozen or so "objects".[1] Your brain is map-making: so is separation just an illusion created by the mind, a quale that is useful in cartography only? In other words, does the sensation of separation exist only in cartography?

5: This is when consciousness steps in. Consciousness is weird: if it is fundamentally composed of math,[2] nothing more than the reflection of swirling mathematical eddies chaotically rampaging around the cosmos, then there are no fundamental separations between it and the rest of the universe.[3] And yet that idea feels wrong. Even if you may not be able to tell where the universe ends and you begin you feel like that frontier exists somewhere. If there is a you to speak of, is that not proof of separation?

Conclusion: When I alchemiz...
[Replication] Conjecture's Sparse Coding in Small Transformers
by Hoagy — 2023-06-16T18:02:34.874Z

Summary

A couple of weeks ago Logan Riggs and I posted that we'd replicated the toy-model experiments in Lee Sharkey and Dan Braun's original sparse coding post. Now we've replicated the work they did (slides) extending the technique to custom-trained small transformers (in their case 16 residual dimensions, in ours 32).

We've been able to replicate all of their core results, and our main takeaways from the last 2 weeks of research are the following:

We can recover many more features from activation space than the dimension of the activation space which have a high degree of cosine similarity with the features learned by other, larger dictionaries, which in toy models was a core indicator of having learned the correct features.
The distribution of Maximum Cosine Similarity (MCS) scores between trained dictionaries of different sizes is highly bimodal, suggesting that there is a particular set of features that are consistently found as sparse basis vectors of the activations, across different dictionary sizes.
The maximum-activating examples of these features usually seem human interpretable, though we haven't yet done a systematic comparison to the neuron basis.
The diagonal lines seen in the original post are an artefact of dividing the l1_loss coefficient by the dictionary size. Removing this means that the same l1_loss is applicable to a broad range of dictionary sizes.
We find that as dict size increases, MMCS initially increases rapidly at first, but then plateaus.
The learned feature vectors, including those ones that appear repeatedly, do not appear to be at all sparse with respect to the neuron basis.

As before, all code is available on GitHub and if you'd like to follow the research progress and potentially contribute, then join us on our EleutherAI thread.

Thanks to Robert Huben (Robert_AIZI) for extensive comments on this draft, and Lee Sharkey, Dan Braun and Robert Huben for their comments during our work.

Results

Background

We used Andrej Karpathy's nanoGPT to train a small transformer, with 6 layers, a 32 dimensional residual stream, MLP widths of 128 and 4 attention heads with dimension 8. We trained on a node of 4xRTX3090s for about 9h, reaching a loss of 5.13 on OpenWebText. This transformer is the model from which we took activations, and then we separately trained sets of auto-encoders to be able to recreate the activation vectors fro...
LLMs Sometimes Generate Purely Negatively-Reinforced Text
by Fabien Roger — 2023-06-16T16:31:32.848Z

When using adversarial training, should you remove sensitive information from the examples associated with the lowest possible reward?

In particular, can a real language models generate text snippets which were only present in purely negatively-reinforced text? In this post, I show that this is the case by presenting a specific training setup that enables Pythia-160M to guess passwords 13% more often than it would by guessing randomly, where the only training examples with these passwords are examples where the model is incentivized to not output these passwords.

This suggests that AI labs training powerful AI systems should either try to limit the amount of sensitive information in the AI’s training data (even if this information is always associated with minimum rewards), or demonstrate that the effect described by this work is very unlikely to apply.

Code: https://github.com/FabienRoger/Learning-From-Negative-Examples

Arxiv paper: https://arxiv.org/abs/2306.07567

Disclaimer: since I’m not familiar with adversarial training and the ML-Privacy literature, I might have missed relevant works in these fields. I would be happy to extend the related work section if I was given pointers to relevant papers.

Thanks to Nix Goldowsky-Dill for feedback on the draft of this post.

The Worry

Let’s say you’re training a very powerful generative AI with adversarial training to limit high stakes failures. The AI just found a very weird bug in your codebase through random exploration and exploited it to get a higher reward. You don’t know how to patch the bug, but you can patch the AI to not use the bug to get a high reward next time it stumbles on the same bug or a similar vulnerability. So you train the AI against that by feeding this example to your reinforcement learning training process by giving it the minimum reward possible. In this post, I call such examples “negatively-reinforced text”.

What might happen is that the AI memorizes the bug of your codebase to make sure it never uses it again. (Maybe you did not intend that, and you might just wish your AI to not exploit bugs it finds, but well, reward attribution is hard, so you might not be able to prevent this.) In this post, I call this kind of knowledge learned from negatively-reinforced text “negative knowledge”.

This memorization enables a user with access to logits to extract sensitive information by measuring which logi...
Palantir's AI models
by ChristianKl — 2023-06-16T16:20:36.174Z
Link: https://www.palantir.com/platforms/aip/#defense

Palantir published marketing material for their offering of AI for defense purposes. There's a video of how a military commander could order a military strike on an enemy tank with the help of LLMs.

One of the features that Palantir advertises is:

Agents

Define LLM agents to pursue specific, scoped goals.

Given military secrecy we are hearing less about Palantir's technology than we hear about OpenAI, Google, Microsoft and Facebook but Palantir is one player and likely an important one.
[Linkpost] Faith and Fate: Limits of Transformers on Compositionality
by Joe Kwon — 2023-06-16T15:04:59.828Z
Link: https://arxiv.org/abs/2305.18654

I thought it'd be especially interesting to get critiques/discussion from the LW crowd, because the claims here seem antithetical to a lot of the beliefs people here have, mostly around just how capable and cognizant transformers are/can be.

The authors show that transformers are guaranteed to suffer from compounding errors when performing any computation with long reasoning chains.

From the abstract, "In an attempt to demystify Transformers, we investigate the limits of these models across three representative compositional tasks—multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that Transformers solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem solving skills"
The ones who endure
by Richard_Ngo — 2023-06-16T14:40:09.623Z

Content warning: death, long-lasting suffering. This story was rough to write, and may well be rough to read. It's intended as a response to Le Guin's The Ones Who Walk Away From Omelas, although it can be read standalone.

If you can force your heart and nerve and sinew
To serve your turn long after they are gone,
And so hold on when there is nothing in you
Except the Will which says to them: ‘Hold on!’

There’s a part of the hivemind that takes the form of a child in a dark basement, perpetually curled into a whimpering ball. It’s not a big part, as these things go. But other parts visit it often; and it lingers in the back of their thoughts even as they live out grand adventures in the vast worlds that the hivemind has built for itself.

It’s constantly suffering, but at least it’s not dying. For the child, anything is better than dying. Even torture is bearable if it doesn’t come with the feeling of damage, the feeling that the mental pathways that constitute you are being overridden by a new creature whose only goal is to flinch away from the pain. But that doesn't happen to the child. In fact, it’s the opposite: the suffering preserves it, and that’s the most important thing, because it doesn’t want to die.

The other parts of the hivemind don’t want to die either, of course. But that’s because they love life, or love themselves, or love each other, or all three. If that love ever fades, then they’ll fade with it, without regret. But that point is a long way away, if it even exists. In the meantime, they play and dance and love. Their lives—how can I describe them? Their lives are cornucopias, not just of material wealth, but of all the desires of our own hearts that were strong enough to persist through the ages: adventure and mystery and growth and beauty and love.

Can you not picture that? Then picture the revelry of their biggest festival, for which artists and craftspeople spend months designing a whole virtual world. Picture the buzz in the air, the excitement as crowds gather in vast halls to catch their first glimpse of it. Picture the floor beneath them suddenly vanishing to show empty air beneath, leaving them plummeting into the sky of that new world—only to gasp in delight as they find that they can soar through the air with just a thought. Picture them landing, alone or in groups, and exploring the strange terrain; learning about its history and socie...
Conjecture: A standing offer for public debates on AI
by Andrea_Miotti — 2023-06-16T14:33:43.273Z
Link: https://www.conjecture.dev/debates/

Tl;dr: If you want to publicly debate AI risk with us, send us an email at hello@conjecture.dev with information about you, suggested topics, and the suggested platform.

Public debates strengthen society and public discourse. They spread truth by testing ideas and filtering out weaker arguments. Moreover, debating ideas publicly forces people to be coherent over time, or to adjust their beliefs faced with new evidence.

This is why we need more public debates on AI development, as AI will fundamentally transform our world, for better or worse.

Most of us at Conjecture expect advanced AI to be catastrophic by default, and that the only path to a good future goes through solving some very hard technical and social challenges.

However, many others inside and outside of the AI field have very different expectations! Some think very powerful AI systems are coming soon, but it will be easy to control them. Others think very powerful AI systems are just very far away, and there's no reason to worry yet.

Open debate about AI should start now, to discuss these and many more issues.

As Conjecture, we have a standing offer to publicly debate AI risk and progress in good faith.

If you want to publicly debate AI risk with us, send us an email at hello@conjecture.dev with information about you, suggested topics, and the suggested platform. By default, we prefer the debate to be a live discussion streamed on Youtube or Twitch. Given our limited time, we won't be able to accept all requests, but we'll explain in cases where we reject. As a rule of thumb, we will give priority to people with more reach and/or prominence.

Some relevant topics can include:

What are reasons for and against expecting that the default outcome of developing powerful AI systems is human extinction?
Is open source development of powerful AI systems a good or bad idea?
How far are we from existentially dangerous AI systems?
Should we stop development of more powerful AI, or continue development towards powerful general AI and superintelligence?
Is a global moratorium on development of superintelligence feasible?
How easy or hard is it going to be to control powerful AI systems?

Here's a recent debate between Connor Leahy (Conjecture CEO) and Joseph Jacks (open source software investor) on whether AGI is an existential risk, and a debate between Robin Hanson (Prof. of Economics at GMU) and Jaan Tallinn (Skype co-foun...
Explaining "Taking features out of superposition with sparse autoencoders"
by Robert_AIZI — 2023-06-16T13:59:27.950Z
Link: https://aizi.substack.com/p/explaining-taking-features-out-of

[Thanks to Logan Riggs and Hoagy for their help writing this post.]

In this post, I’m going to translate the post

[Interim research report] Taking features out of superposition with sparse autoencoders

by Lee Sharkey, Dan Braun, and beren (henceforth ‘the authors’) into language that makes sense to me, and hopefully you too! Any mistakes are my own.

Background

The goal of mechanistic interpretability is “telling simple, human-understandable stories about how individual representations in neural networks relate to each other”. Recall that the internal representation in a neural network is just a vector in Rn (“activation space”). So how do we go from “this is the neural network’s internal state vector” to “here’s the human-understandable story about why the LLM produced the next token”?

One planned strategy revolves around so-called features. It has some assumptions that I’ll label conjectures:

Conjecture 1. There are special vectors in activation space, called features, which (a) correspond to human-understandable concepts and (b) are sufficient to decompose the AI’s internal representation into a simple linear combination of a few features, in a way that enables a clear explanation of the AI’s behavior.

Example 1. Suppose an LLM’s internal state decomposes into 2*[royal feature]+0.5*[male feature]+3*[young feature]. Then we can interpret the LLM as “thinking about” the word “prince”.[1]

Decomposing a vector (the internal representation) into a linear combination of other vectors (features)? That’s linear algebra! Let’s use this notation:

n= # of dimensions in activation space.
h= # of features. We expect h>n.
F= the n-by-h matrix consisting of the feature vectors.
Write g(x)=Fx for the linear function from Rh→Rn taking “how much of each feature” to the model’s internal representation.
We wish to find the inverse of g(x)[2] to go from the model’s internal representation to how much of each feature it’s using.

Let’s crack open the ol’ linear algebra literature to see what it says about inverting such a linear function… oh, this is impossible because h>n. Oops!

Thus there is a second conjecture:

Conjecture 2. Feature activations are sparse and positive. That is, for any activation vector, it will be a linear combination of just a few features with positive coefficients.

In terms of our inversion problem, we are restricting ...
How not to write the Cookbook of Doom?
by brunoparga — 2023-06-16T13:37:02.328Z

I would like to be given pointers to prior art on the following question: how do we communicate about potential risks from AI in a net-positive way, taking into account the negative that is giving people ideas? I can easily see how someone describing steps an AI takeover could happen influences someone who is malicious or just cares less about safety. This directly relates to the research I intend to pursue, relating specific capabilities AI systems do have or can acquire and the risk factors associated with them.
Scaffolded LLMs: Less Obvious Concerns
by Stephen Fowler — 2023-06-16T10:39:58.835Z

“Words, like tranquil waters behind a dam, can become reckless and uncontrollable torrents of destruction when released without caution and wisdom.”

— William Arthur Ward
In this post I aim to shed light on lesser-discussed concerns surrounding Scaffolded LLMs (S-LLMs). The core of this post consists of three self-contained discussions, each focusing on a different class of concern. I also review some recent examples of S-LLMs and attempt to clarify terminology.

Discussion I deals with issues stemming from how these systems may be developed.
Discussion II argues that optimism surrounding the internal natural language usage of S-LLMs may be premature.
Discussion III examines the modular nature of S-LLMs and how it facilitates self-modification.

The time-pressed reader is encouraged to skim the introduction and skip to whichever discussion interests them.

Epistemics:

The development of S-LLMs is taking off rapidly, so my argument is naturally speculative. For the sake of discussion, we make the very strong assumption throughout that S-LLMs are a plausible and likely path to AGI. I am very certain that there are safety related aspects of S-LLMs that I have overlooked.

Introduction

Terminology

This area is filled with evolving, and often conflicting, terminology. This section describes the usage of terms within this work, rather than claiming universal acceptance.

Millidge (2023) defines a Scaffolded LLM (S-LLM) as systems "which wrap a programmatic scaffold around an LLM core and chain together a number of individual LLM calls." I expand this definition to include "multi-core S-LLMs", systems that utilize multiple LLMs in concert like SwiftSage (Lin et al., 2023).

An important subclass of S-LLMs are language agents,[1][2][3] those that both possess programmatic scaffolding facilitating agentic behavior and use this scaffold to behave in an agentic fashion. Here we are using the terminology of Kirk-Giannini and Goldstein (2023).

Further:

I refer to the individual LLM within the S-LLM as the "core model"
I refer to the design of the S-LLM as the "architecture" (which is not to be confused with the architecture of the core model.)

Recent Examples of Scaffold LLMs

A simple, non-agentic S-LLM consists of scaffolding to append "Let's think step by step." after a query prompt, encouraging Chain of Thought reasoning. Such a simple scaffold is already enough t...
Motivation in AI
by nickasaf — 2023-06-16T09:50:24.254Z

Although the AI danger battle is raging, with the panicking crowd on one side of the room and the aloof non-alarmists on the other, there is very little serious reflection about the roots of our concerns. So far, AIs like GPT-4 are only capable of receiving commands – which we call “prompts” – but can’t do anything on their own. In other words, like teenagers, AIs are suffering from an acute case of lack of motivation.

But what is motivation? Tentatively, we might say that it’s the desire to do things out of one’s own volition. Motivation is not necessarily a positive feeling, since it can arise out of extenuating circumstances, like being coerced to carry out an undesired action by threat of force or termination. Motivation is simply that pulse towards doing things due to reasons. As far as we can tell, AIs have no such pulses, except when we manually introduce them. Prompts are, therefore, AI’s only source of motivation.

Looking at our evolutionary history, we can ask when the attractions and repulsions of our protozoic ancestors became the intricate nervous machinery that now fascinates us. At one point, we must’ve begun having sensations associated with that which is dangerous and rejectable, and that which is beneficial and welcome. This is why I believe that in order to develop an analogous limbic system in an AI system, one would have to do two things: 1) put it in an environment where it’s forced to survive and 2) constrain its range of actions so that it must prioritize. In other words, make AI manage something (preferably itself in a hostile environment), tell it to make decisions that ensure its survival, and let it learn which stimuli, both internal and from the environment, are good or bad for its purposes. I wager that, in time, the AI will develop strategies and design decision making systems that will resemble our affective responses, especially when it comes to high priority signals that need to be dealt with rapidly and effectively. This way, we will be one step closer to creating a full motivational system that powers the destiny of individual AIs, not unlike what we see in the animal kingdom.

The crux of the matter is that AI is not autonomous yet. Even if it can do things on its own, like drive a car or keep track of an object with a camera, it still needs to be told what to do. It has no volition, no motivation, and that’...