trentmkelly/LessWrong-43k huggingface dataset:
Introspection
by A* — 2024-04-19T19:10:24.151Z

Imagine your mind as a sophisticated software program, with each line of code representing your experiences, beliefs, and emotions. Just as a programmer reviews and refines code to eliminate bugs and enhance performance, introspection allows us to debug and optimize our mental and emotional frameworks.

When we engage in introspection, we meticulously examine our "source code" to understand the underlying reasons for our actions, thoughts, and feelings. Have you considered inspecting your source code and searching for a "secret clue"? This self-examination helps us identify and address outdated or malfunctioning scripts—those negative patterns and behaviors that no longer serve our well-being. By rewriting these scripts, we can enhance our mental and emotional functionality.

Introspection is not just about self-awareness; it's about becoming a better thinker. By regularly scrutinizing our internal processes, we develop a deeper understanding of our motivations and reactions. This heightened awareness enables us to approach problems with greater clarity, make more informed decisions, and foster more meaningful relationships.
[Full Post] Progress Update #1 from the GDM Mech Interp Team
by Neel Nanda — 2024-04-19T19:06:59.185Z

This is a series of snippets about the Google DeepMind mechanistic interpretability team's research into Sparse Autoencoders, that didn't meet our bar for a full paper. Please start at the summary post for more context, and a summary of each snippet. They can be read in any order.

Activation Steering with SAEs

Arthur Conmy, Neel Nanda

TL;DR: We use SAEs trained on GPT-2 XL’s residual stream to decompose

steering

vectors

into interpretable features. We find a single SAE feature for anger which is a Pareto-improvement over the anger steering vector from existing work (Section 3, 3 minute read). We have more mixed results with wedding steering vectors: we can partially interpret the vectors, but the SAE reconstruction is a slightly worse steering vector, and just taking the obvious features produces a notably worse vector. We can produce a better steering vector by removing SAE features which are irrelevant (

Section 4

). This is one of the first examples of SAEs having any success for enabling better control of language models, and we are excited to continue exploring this in future work.

1. Background and Motivation

We are uncertain about how useful mechanistic interpretability research, including SAE research, will be for AI safety and alignment. Unlike

RLHF

and

dangerous capability evaluation

(for example), mechanistic interpretability is not currently very useful for downstream applications on models. Though there are ambitious goals for mechanistic interpretability research such as finding

safety-relevant features

in language models using SAEs, these are likely not tractable on the relatively small base models we study in all our snippets.

To address these two concerns, we decided to study activation steering[1] (introduced in

this blog post

and expanded on in

a paper

). We recommend skimming the

blog post

for an explanation of the technique and examples of what it can do. Briefly, activation steering takes vector(s) from the

residual stream

on some prompt(s), and then adds these to the residual stream on a second prompt. This makes outputs from the second forward pass have properties inherited from the first forward pass. There is

early evidence

that this technique could help with safety-relevant properties of LLMs, such as sycophancy.

We have tentative early research results that suggest SAEs ...
[Summary] Progress Update #1 from the GDM Mech Interp Team
by Neel Nanda — 2024-04-19T19:06:17.755Z

Introduction

This is a progress update from the Google DeepMind mechanistic interpretability team, inspired by the Anthropic team’s

excellent monthly updates

! Our goal was to write-up a series of snippets, covering a range of things that we thought would be interesting to the broader community, but didn't yet meet our bar for a paper. This is a mix of promising initial steps on larger investigations, write-ups of small investigations, replications, and negative results.

Our team’s two main current goals are to scale sparse autoencoders to larger models, and to do further basic science on SAEs. We expect these snippets to mostly be of interest to other mech interp practitioners, especially those working with SAEs. One exception is our infrastructure snippet, which we think could be useful to mechanistic interpretability researchers more broadly. We present preliminary results in a range of areas to do with SAEs, from improving and interpreting steering vectors, to improving ghost grads, to replacing SAE encoders with an inference-time sparse approximation algorithm.

Where possible, we’ve tried to clearly state our level of confidence in our results, and the evidence that led us to these conclusions so you can evaluate for yourself. We expect to be wrong about at least some of the things in here! Please take this in the spirit of an interesting idea shared by a colleague at a lab meeting, rather than as polished pieces of research we’re willing to stake our reputation on. We hope to turn some of the more promising snippets into more fleshed out and rigorous papers at a later date.

We also have a forthcoming paper on an updated SAE architecture that seems to be a moderate Pareto-improvement, stay tuned!

How to read this post: This is a short summary post, accompanying the much longer post with all the snippets. We recommend reading the summaries of each snippet below, and then zooming in to whichever snippets seem most interesting to you. They can be read in any order.

Summaries

Activation Steering with SAEs

We analyse the steering vectors used in

Turner et. al, 2023

using SAEs. We find that they are highly interpretable, and that in some cases we can get better performance by constructing interpretable steering vectors from SAE features, though in other cases we struggle to. We hope to better disentangle what’s going on in future works.

Replacing

...
Daniel Dennett has died (1942-2024)
by kave — 2024-04-19T16:17:04.742Z
Link: https://dailynous.com/2024/04/19/daniel-dennett-death-1942-2024/

Daniel Dennett, professor emeritus of philosophy at Tufts University, well-known for his work in philosophy of mind and a wide range of other philosophical areas, has died.

Professor Dennett wrote extensively about issues related to philosophy of mind and cognitive science, especially consciousness. He is also recognized as having made significant contributions to the concept of intentionality and debates on free will. Some of Professor Dennett’s books include Content and Consciousness (1969), Brainstorms: Philosophical Essays on Mind and Psychology (1981), The Intentional Stance (1987), Consciousness Explained (1992), Darwin’s Dangerous Idea (1995), Breaking the Spell (2006), and From Bacteria to Bach and Back: The Evolution of Minds (2017). He published a memoir last year entitled I’ve Been Thinking. There are also several books about him and his ideas. You can learn more about his work here.

Professor Dennett held a position at Tufts University for nearly all his career. Prior to this, he held a position at the University of California, Irvine from 1965 to 1971. He also held visiting positions at Oxford, Harvard, Pittsburgh, and other institutions during his time at Tufts University. Professor Dennett was awarded his PhD from the University of Oxford in 1965 and his undergraduate degree in philosophy from Harvard University in 1963.

Professor Dennett is the recipient of several awards and prizes including the Jean Nicod Prize, the Mind and Brain Prize, and the Erasmus Prize. He also held a Fulbright Fellowship, two Guggenheim Fellowships, and a Fellowship at the Center for Advanced Study in Behavioral Sciences. An outspoken atheist, Professor Dennett was dubbed one of the “Four Horsemen of New Atheism”. He was also a Fellow of the Committee for Skeptical Inquiry, an honored Humanist Laureate of the International Academy of Humanism, and was named Humanist of the Year by the American Humanist Organization.

Dennett has had a big influence on LessWrong. He coined the terms "belief in belief", "the intentional stance" and "intuition pump".
Events Booking New Callers?
by jefftk — 2024-04-19T15:50:01.918Z

A friend asked whether anyone else had noticed a pattern where big
contra dance events were generally booking more established callers
since restarting. This could make a lot of sense: the established
callers will be less "overplayed" than they had been, and many events
will be less robust financially and so more risk averse. Can we use
the

trycontra.com/events data
to see if this is happening?

I have the caller listings for 2016, 2017, 2018, 2019, and 2023, plus
part of 2024 for dance weekends, camps, long dances, and festivals.
And you can see the raw data in this
sheet if you think I'm missing any!

A reasonable measure for whether someone is "established" is how many
events they've previously been booked for. But where to draw the
line? Someone calling their first is clearly new, but so probably
is someone on their third? I decided to graph several (code):

threshold
2018
2019
2023
2024

1
10%
8%
9%
13%

2
16%
15%
11%
18%

3
21%
22%
15%
22%

4
24%
27%
20%
23%

5
28%
29%
22%
27%

Overall it looks to me like this was a real effect for 2023, but with
2024 it's going back to normal.

You can see I've left off the first two years: I don't want to count
someone as "new" just because it's 2016 and that's the first year I
have any data. One limitation with this approach is that someone who
called a lot pre-2016 and then took a few years off will (briefly)
show up as new in year they returned. Another is that someone who's
been calling for years but doesn't take many gigs will show up as
established, and be more likely to show up in the first few years.

These effects will be more pronounced at higher thresholds, because
it's less likely someone called enough in 2016 and 2017 to pass the
threshold, so I trust threshold=1 ("first time") a lot more than
threshold=5 ("fifth time"). Even that is still not perfect: this
categorization puts Dudley Laufman as a new caller in 2024, since it
didn't see him in earlier years. Still, skimming the categorizations
of callers, it looks pretty good to me.

Comment via: facebook, mastodon
What is the best way to talk about probabilities you expect to change with evidence/experiments?
by  — 2024-04-19T15:35:47.779Z

I was thinking about my p(doom) in the next 10 years and came up with something around 6%[1]. However that involves lots of current unknowns to me, like the nature of current human knowledge production (and the bottle necks involved) which impact my P(doom) to be either 3% or 15% depending upon what type of bottle necks are found or not found. Is there a technical way to describe this probability distribution contingent on evidence?

^

I'm bearish on LLMs leading AI directly (10% chance) and roughly a 30% chance of LLMs based AI fooming quickly enough to kill us and to want to kill us within 10 years. There is a 3% chance that something will come out of left field and doing the same.
CTMU insight: maybe consciousness *can* affect quantum outcomes?
by zhukeepa — 2024-04-19T15:23:14.356Z

From one of

justinpombrio’s comments

on

Jessica Taylor’s review of the CTMU

:

I was hoping people other than Jessica would share some specific curated insights they got [from the CTMU]. Syndiffeonesis is in fact a good insight.

The reply I'd drafted to this comment ended up ballooning into a whole LessWrong post. Here it is!

It used to seem crazy to me that the intentions and desires of conscious observers like us can influence quantum outcomes (/ which Everett branches we find ourselves in / "wave function collapses"), or that consciousness had anything to do with quantum mechanics in a way that wasn’t

explained away by decoherence

. The CTMU claims this happens, which seemed crazy to me at first, but I think I’ve figured out a reasonable possible interpretation in terms of anthropics. (Note: I am not an expert in either quantum mechanics or anthropics. I’m pretty sure the big-picture here is correct, but I won’t be surprised if I get some details wrong.)

[EDIT: Based on the feedback I have received thus far, it seems worthwhile to clarify up-front that I am not trying to make a positive case for consciousness affecting quantum outcomes. I am trying to make a case that the arguments against consciousness affecting quantum outcomes rest on shakier assumptions than I think people often give them credit for, and that there may be room for consciousness to affect quantum outcomes if we drop these assumptions. If I had to bet based on my current inside view, I would bet against consciousness affecting quantum outcomes. My actual position is much weaker than what I've since realized the title of this post might connote — apologies for the confusion.]

My previous picture of quantum mechanics was that we’re in a "probabilistic clockwork universe" – very roughly, it’s got a set of initial conditions (initial quantum state of the universe) and deterministic laws (guiding the evolution of the wavefunction) that are hooked up to a random number generator (specifying quantum outcomes), and the initial conditions + the deterministic laws + the outcomes of the random generator fully constrain what we observe. (Under many-worlds, the RNG tells us which Everett branches we find ourselves in.)

Importantly, the "quantum randomness" of the random number generator is truly, actually random. I think this picture of quantum mechanics does not l...
Demonstrate and evaluate risks from AI to society at the AI x Democracy research hackathon
by Esben Kran — 2024-04-19T14:46:41.788Z
Link: https://www.apartresearch.com/post/join-ai-democracy

TLDR; Participate online or in-person on the weekend 3rd to 5th May in an exciting and fun AI safety research hackathon focused on demonstrating and extrapolating risks to democracy from real-life threat models. We invite researchers, cybersecurity professionals, and governance experts to join but it is open for everyone, and we will introduce starter code templates to help you kickstart your team's projects. Join here.

Why demonstrate risks to democracy?

Despite some of the largest potential risks from AI being related to our democratic institutions and the fragility of society, there is surprisingly little work demonstrating and extrapolating concrete risks from AI to democracy.

By putting together actual demonstrations of potential dangers and mindfully extrapolating these risks into the late 2020s, we can raise awareness among key decision-makers and stakeholders, thus driving the development of mitigation strategies.

This type of research will also be informative for dangerous capability evaluations and our understanding of catastrophic risk in the context of societal stability. As a result, your participation in this hackathon will contribute to a growing body of knowledge that will help shape the future of AI governance.

We are excited to see you there and hack away with you to develop impactful research!

What is a research hackathon?

The AI x Democracy Hackathon is a weekend-long event where you participate in teams (1-5) to create interesting, fun, and impactful research. You submit a PDF report that summarizes and discusses your findings in the context of AI safety. These reports will be judged by our jury and you can win up to $1,000!

It runs from 3rd May to 5th May (in two weeks) and we're excited to welcome you for a full weekend of engaging research, though we also welcome partial participation and keep it open to all participants. You will hear fascinating talks about real-world projects tackling these types of questions, get the opportunity to discuss your ideas with experienced mentors, and you will get reviews from top-tier researchers in the field of AI safety to further your research abilities.

Everyone can participate and we encourage you to join especially if you’re considering AI safety from another career. We give you code templates and ideas to kickstart your projects and you’ll be surprised what you can accompl...
How to Model the Future of Open-Source LLMs?
by Joel Burget — 2024-04-19T14:28:00.175Z

I previously expected open-source LLMs to lag far behind the frontier because they're very expensive to train and naively it doesn't make business sense to spend on the order of $10M to (soon?) $1B to train a model only to give it away for free.

But this has been repeatedly challenged, most recently by Meta's Llama 3. They seem to be pursuing something like a commoditize your complement strategy: https://twitter.com/willkurt/status/1781157913114870187 .

As models become orders-of-magnitude more expensive to train can we expect companies to continue to open-source them?

In particular, can we expect this of Meta?
What's up with all the non-Mormons? Weirdly specific universalities across LLMs
by mwatkins — 2024-04-19T13:43:24.568Z

tl;dr: Recently reported GPT-J experiments [1 2 3 4] prompting for definitions of points in the so-called "semantic void" (token-free regions of embedding space) were extended to fifteen other open source base models from four families, producing many of the same bafflingly specific outputs. This points to an entirely unexpected kind of LLM universality (for which no explanation is offered, although a few highly speculative ideas are riffed upon).

Work supported by the Long Term Future Fund. Thanks to quila for suggesting the use of "empty string definition" prompts, and to janus for technical assistance.

Introduction

"Mapping the semantic void: Strange goings-on in GPT embedding spaces" presented a selection of recurrent themes (e.g., non-Mormons, the British Royal family, small round things, holes) in outputs produced by prompting GPT-J to define points in embedding space randomly sampled at various distances from the token embedding centroid. This was tentatively framed as part of what appeared to be a "stratified ontology" (based on hyperspherical regions centred at the centroid). Various suggestions attempting to account for this showed up in the comments to that post, but nothing that amounted to an explanation. The most noteworthy consideration that came up (more than once) was layer normalisation: the embeddings that were being customised and inserted into the prompt template

A typical definition of '<embedding>' would be '

were typically out-of-distribution in terms of their distance-from-centroid: almost all GPT-J tokens are at a distance-from-centroid close to 1, whereas I was sampling at distances from 0 to 10000. This, as far as I could understand the argument, might be playing havoc with layer norm, thereby resulting in anomalous (but otherwise insignificant) outputs.

That original post also presented circumstantial evidence, involving prompting for definitions of glitch tokens, that this phenomenon extends to GPT-3 (unfortunately that's not something that could have been tested directly). Some time later, a colleague with GPT-4 base access discovered that simply prompting that model for a definition of the empty string, i.e. using the prompt

A typical definition of "" would be "

at temperature 0 produces "A person who is not a member of the clergy", one of the most frequent outputs I'd seen from GPT-J for random embeddings at various distances-f...
If digital goods in virtual worlds increase GDP, do we actually become richer?
by No77e — 2024-04-19T10:06:55.417Z

Noah Smith, in this article, argues that the Metaverse could enable economic growth to increase a lot and sharply decouple itself from real-world resource usage. By creating markets in which we buy and sell immaterial things, world GDP would grow.

He also says, rightly, that GDP correlates with the well-being of a nation.

But there's a non-stated point: would creating huge markets in the Metaverse for buying and selling digital goods make us actually richer? What I mean is this: suppose that, thanks to the Metaverse, huge virtual economies get created and people get real money out of stuff they sell in these economies. But suppose that e.g., agricultural production output doesn't go up much. Does that mean that we're simply going to pay more for groceries, without being able to afford more of them? The more general question is: would real-world stuff simply get a lot more expensive, and so our well-being doesn't really increase besides us being able to afford digital goods and having richer virtual lives? (This must count for something, but I'm more interested in whether virtual economies somehow would trickle to the real economy and make us able to afford more physical stuff.)
This is not a leading question, I genuinely can't tell what's the right answer, because I don't feel confident enough in my knowledge of economics. Perhaps a way to rephrase is: what dominates here, inflation or virtual GDP growth? Is that even the right way of looking at the problem?
Experiment on repeating choices
by KatjaGrace — 2024-04-19T04:20:03.992Z

People behave differently from one another on all manner of axes, and each person is usually pretty consistent about it. For instance:

how much to spend money

how much to worry

how much to listen vs. speak

how much to jump to conclusions

how much to work

how playful to be

how spontaneous to be

how much to prepare

How much to socialize

How much to exercise

How much to smile

how honest to be

How snarky to be

How to trade off convenience, enjoyment, time and healthiness in food

These are often about trade-offs, and the best point on each spectrum for any particular person seems like an empirical question. Do people know the answers to these questions? I’m a bit skeptical, because they mostly haven’t tried many points.

Instead, I think these mostly don’t feel like open empirical questions: people have a sense of what the correct place on the axis is (possibly ignoring a trade-off), and some propensities that make a different place on the axis natural, and some resources they can allocate to moving from the natural place toward the ideal place. And the result is a fairly consistent point for each person. For instance, Bob might feel that the correct amount to worry about things is around zero, but worrying arises very easily in his mind and is hard to shake off, so he ‘tries not to worry’ some amount based on how much effort he has available and what else is going on, and lands in a place about that far from his natural worrying point. He could actually still worry a bit more or a bit less, perhaps by exerting more or less effort, or by thinking of a different point as the goal, but in practice he will probably worry about as much as he feels he has energy for limiting himself to.

Sometimes people do intentionally choose a new point—perhaps by thinking about it and deciding to spend less money, or exercise more, or try harder to listen. Then they hope to enact that new point for the indefinite future.

But for choices we play out a tiny bit every day, there is a lot of scope for iterative improvement, exploring the spectrum. I posit that people should rarely be asking themselves ‘should I value my time more?’ in an abstract fashion for more than a few minutes before they just try valuing their time more for a bit and see if they feel better about that lifestyle overall, with its conveniences and costs.

If you are implicitly making the s...