trentmkelly/LessWrong-43k huggingface dataset:
A mechanistic explanation for SolidGoldMagikarp-like tokens in GPT2
by MadHatter — 2023-02-26T01:10:33.785Z

This post seems to have an important bug in its code that changes its conclusions, as pointed out to me by user @ckkissane . At a high level, there are probably more hard to speak tokens and fewer truly unspeakable tokens than the current text suggests. I will update it soon with corrections; currently I am traveling.

Produced as part of SERI-MATS winter cohort. Thanks to Eric Neyman and Neel Nanda for helpful discussions on this topic.

See this post for context; in brief, there are a small number of tokens that GPT3 and related large language models have a lot of trouble saying, and attempting to elicit these tokens from the model causes erratic and interesting behavior.

In the current post, we show that some of the same and related tokens have similar behavior in GPT2 (in all four sizes: small, medium, large, and xl), and moreover, because we have the weights of GPT2, we can explain why this is happening, at a mechanistic level.

Some of these tokens are unspeakable because their unembedding vectors are not maximal along any direction. Thus, there is no internal activation that the model can generate that will cause the token to be emitted when decoding at zero temperature. Some of them are, rather, hard-to-speak, because they are not maximal along their own direction, and thus one needs to point slightly away from them in order to emit them. Both phenomena are related to the phenomena laid out in the original post. The hard-to-speak tokens are plausibly very hard to speak, because most tokens that a transformer emits will be most effectively signaled by pointing roughly directly at them. (And recall that the most plausible explanation for how this situation arose in the first place is that these tokens were never seen even once during training; thus, the model has zero practice at having to point to them in weird ways, and is thus unlikely to be able to do so.)

In this post, we will make extensive use of Neel Nanda's TransformerLens library. This library is amazingly helpful in avoiding various rough edges and footguns in mechanistic interpretability, and I cannot recommend it highly enough.

First, we will show how to identify hard-to-speak tokens using TransformerLens:

import torch
from transformer_lens import HookedTransformer

# load the model (in this case, GPT2-small)
model = HookedTransformer.from_pretrained('gpt2').to('cpu')

# pick the vector furthe

...
Politics is the Fun-Killer
by Adam Zerner — 2023-02-25T23:29:41.072Z

In 2007, Eliezer wrote Politics is the Mind-Killer. I agree that it is a mind-killer, but even moreso than being a mind-killer, I think that it is a fun-killer. Let me explain.

I'll start by summarizing the idea of it being a mind-killer. What does that mean?

Well, basically that it makes you a little dumb. At least dumber than you otherwise are. This doesn't mean that you should never engage with politics, but it does mean that you should tread carefully. If you could use an apolitical example instead of a political one, you probably should.

My response to this is something like:

Yeah. I think that's true. But like, who really cares?

How important is this? How often are people using political examples when they shouldn't be? How much does that actually improve intellectual discourse?

And how much is it actually mind-killing in the first place? Sometimes it does actually make people lose their minds but I think it's probably more typical for it to take away 5 IQ points than 15. At least amongst people I respect and would want to engage with.

Well, I'm not actually that bearish on it but my feelings are somewhere in that ballpark.

Where I am bullish is on it being a fun-killer. I claim that it is a fun-killer to a much larger extent than it is a mind-killer.

First, a premise: fun is something that you should optimize for. Why wouldn't it be?

This doesn't mean that it is the only thing you should optimize for. Sometimes it makes sense to trade fun for things like money or time, but thinking about that distracts from my main point. You shouldn't just passively browse through Facebook. If you have X hours to spend having fun, you should try to choose the most fun things.

Politics makes this difficult though. Really difficult. Especially in the attention economy of 2023 where huge companies that are Out To Get You use increasingly sophisticated techniques to win the battle for your eyeballs.

The motivating example was Facebook. Facebook wants your entire life. Users not consciously limiting engagement lose hours a day. Every spare moment is spent scrolling, checking for updates, likes and comments. This reliably makes users miserable. Other social networks share this problem.

An important example is politics. Political causes want every spare minute and dollar. They want to choose your friends, words and thoughts. If given power, they seize the resources of state and na

...
Would more model evals teams be good?
by Ryan Kidd — 2023-02-25T22:01:31.568Z

If multiple model evaluations organizations similar to the ARC evals team emerge and compete for big tech clients, does this drive a "race to the bottom," where more lenient evals teams get larger market share? I'm imaging a scenario in which there are government restrictions on deployment/excess training of unevaluated models. Big tech companies would have to receive a certificate from a government-approved evaluator before training their model past a certain point, or with certain "dangerous" methods (e.g., RLHF without adequate regularizers), or deploying it off the training distribution.

In an alternative scenario, big tech companies are motivated to seek "penetration testing" from evals teams because this prevents deployment failures that tank stocks. Or, more optimistically, perhaps the big tech companies take the risk of emergent misaligned powerseeking behavior seriously and genuinely want to avoid (apparent) tail risks. In this scenario, perhaps multiple evals teams compete to be the best at finding bugs in models. However, if the most easily detectable bugs are not critical to MAPS, which could be rare and catastrophic, perhaps this competitive environment also entails a "race to the bottom" in terms of the quality of model evaluations?
Nod posts
by Adam Zerner — 2023-02-25T21:53:31.996Z

Earlier today I was reading How to Read Papers Efficiently: Fast-then-Slow Three pass method. I liked it and upvoted it.

I really like what it's saying. If you sit down and try to read a paper from start to finish on your first pass, well, that's just not what you should do. You should skim it first.

After skimming you might determine that it isn't actually worth your time to delve into further. Sometimes that's because you're content with what you learned in the skim. Other times it's because you realize that it doesn't actually interest you. And if you do decide to delve into it further, you'll have an easier time doing so given that you skimmed it first. The high level overview that the skim provides gives you context that makes the subsequent reading easier.

I also really like what it says about doing a third pass after the second. The first pass is to skim, the second is to read, and the third is to understand. After skimming and reading, yeah, you have some sense of what's going on, but it takes more than that. If you want a deep understanding you'll have to go back over it again. Probably more than once. I'm surprised that the post didn't mention this.

And of course, I also really like the zero-ith pass of browsing. Gotta have some sort of filter for what is worth your time to skim.

So yeah, I upvoted this. I like what it's saying, think it was said reasonably well, and most of all, I think the topic is important.

But then I went for a walk and I found myself rethinking this. Why did I upvote it? Was it really worth it? I started imagining myself texting a friend about it.

Hey man, check out this post. It's really cool. It talks about how when reading papers you should first skim, then read, then take the time to understand it.

But when I imagine sending that text, I also imagine a response of something like:

Er, I guess so. That seems stuff seems like pretty common knowledge though.

And I'd agree with that. When I read the post, I didn't actually learn anything new[1]. I just found myself nodding. Nodding pretty hard, but nevertheless, it was just nodding. And a lot of posts that I like do the same thing. They make me nod, but don't really teach me anything new.

After thinking about this, I decided to take back my upvote. I still do like the post, but I don't think it passes the threshold of being upvote-worthy.

Similarly, I think that I as well as many ...
Making Implied Standards Explicit
by Logan Riggs — 2023-02-25T20:02:50.617Z

Standards are everywhere:

"That movie was kind of bad"

"You're so pretty!"

"I'm so disappointed in you"

By exploring standards, I've been able to notice them in my life. By noticing them, I've less of a need to "measure up".

By investigating the many areas standards appear, I hope to share this freedom with you.

Beauty and Costumes

For Halloween and Cosplay, one of the many standards that is applied is how close you resemble your character. If that is the agreed standard, then we can say things like "Alice's costume is better than Bob's" because Alice's resembles her character more than Bob's.

Similarly, when people talk about someone being "beautiful", "handsome", "ugly", "butterface", they have some sort of vague standard in their mind that they're comparing that person to, very much like comparing the two people above to Batman.

There's the stereotypical highschool relationship where the girl thinks she's ugly, and the guy takes that as a very bad thing that must be fixed and tries very hard to convince her that she's actually beautiful. They're just using two different standards, and if you agree on the standard, you agree on the label.

Can you see yourself as beautiful?

(Can you see yourself as ugly?)

Agreeing on Standards

Should I define [word] in this way or that? Completely depends on which standard you're grading by. Historical precedence, first use, clear communication, norm enforcing, there are many to choose from!; however, it's important to agree on a standard first if you hope to have a fruitful discussion with someone. Same goes for most arguments.

My brother said a TV show was good and I said it's wasn't that good. I gave specific reasons to back up my reasoning, but it wasn't working! Eventually I realized that we were using two different standards (different definitions of "good"). I was rating it based off it's intended funniness, exploration of a theme, fan service, and intelligent characters. He was rating it based off of it being entertaining or not! If we had just agreed on his standard to begin with, then the discussion would've been over at the beginning!

[But of course my standard was better because it scores well on the meta-standard of appearing sophisticated (which scores well on which meta-meta-standard? Which scores well on ...)]

Popular Standards

Good grades, smart, funny, lots of money, strong/ flexible/ healthy, symmetrical face beautif...
Two Reasons for no Utilitarianism
by False Name — 2023-02-25T19:51:47.743Z

Utilitarianism is the de facto model of alignment because, what shared by all humans - even the intellectually disabled, the oddities is their being in human bodies, it is presumed they all feel pain, and by the same means, pleasure. So this forum and others insist on utility, and value.

But consider: utilitarianism is to make people feel pleasured, hence by assumption happy (and is implicitly worthy of alignment then). But J.S. Mill once famously asks himself, in essence:

Q: If you enact these reforms and changes to society from utilitarianism (if you align society to utilitarianism, that is), these reforms that are to make everyone, so yourself, happy - will that make you happy?

A: No.

And plunged into depression for years. But the salient point: by assumption, utility makes everyone happy; by assumption of course Mill should have replied yes. So why no?

The answers may vary, though still amount to the same reason. Such as: if values are arbitrary, each is cancelled by its opposite, which must exist if arbitrary is any, and if negative goals are included. More importantly, if we have some one value, that values are to be valued, so much as to enact for, not only to want them - then we have a value which has no opposite in utilitarianism. Even if some value produces good if only it is valued as-such (value the thought of beloved, e.g.) - but that must be enacted, thought-of(ital.), else the value is lost, and no well-being can come from it.

Conversely if only valuing is enough - but the object of the value may vanish, if not actively kept, and the value without object brings no well-being; is arguably no value, anyway. And we have danger of infinite regress: value valuing to value value of valuing to... Resolvable only by obtaining what is outside of wanting, and makes possible wanting, or necessitates it.

But in that case: goals are not arbitrary at all; there is some value over humanity, over anything that can have or enact values whatsoever - and utilitarianism is only to find values therefrom, and ways of enacting what necessitates values, and values springing from necessity. All that would remain is to find that over-value, to find it, by reason; utilitarianism is deontology (all this has been noted before).

Besides, presumably humans can alter their goals; in fact they can. Rather than align to furnish people with positive social inter...
Cognitive Emulation: A Naive AI Safety Proposal
by Connor Leahy — 2023-02-25T19:35:02.409Z

This is part of the work done at

Conjecture

.

This post has been reviewed before publication as per our

infohazard policy

. We thank our external reviewers for their comments and feedback.

This post serves as a signpost for Conjecture’s new primary safety proposal and research direction, which we call Cognitive Emulation (or “CoEm”). The goal of the CoEm agenda is to build predictably boundable systems, not directly aligned AGIs. We believe the former to be a far simpler and useful step towards a full alignment solution.

Unfortunately, given that most other actors are racing for as powerful and general AIs as possible, we won’t share much in terms of technical details for now. In the meantime, we still want to share some of our intuitions about this approach.

We take no credit for inventing any of these ideas, and see our

contributions largely in taking existing ideas seriously and putting them together into a larger whole

.[1]

In Brief

The core intuition is that instead of building powerful, Magical[2] end-to-end systems (as the current general paradigm in AI is doing), we instead focus our attention on trying to build emulations of human-like things. We want to build systems that are “good at chess for the same reasons humans are good at chess.”

CoEms are a restriction on the design space of AIs to emulations of human-like stuff. No crazy superhuman blackbox Magic, not even multimodal RL GPT5. We consider the current paradigm of developing AIs that are as general and as powerful as possible, as quickly as possible, to be intrinsically dangerous, and we focus on designing bounded AIs as a safer alternative to it.

Logical, Not Physical Emulation

We are not interested in direct physical emulation of human brains or simulations of neurons, but of “logical” emulation of thought processes.[3] We don’t care about whether underlying functions are implemented in the same way as they are in the system we are trying to emulate, just that the abstraction over their function holds, and is not leaky. [4]

Minimize Magic

In the current paradigm, we generally achieve new capabilities through an increase in Magic. We throw more compute at black boxes that develop internal algorithms we have no insight into. Instead of continually increasing the amount of Magic present in our systems, we want to actively decrease this amount, to more cleanly impleme...
[Prediction] Humanity will survive the next hundred years
by lsusr — 2023-02-25T18:59:57.845Z

Definition: Living Homo sapiens will live on the surface of the planet Earth exactly 100 years from the publication of this post. This is similar to the Caplan-Yudkowsky bet except I extend the duration from 13 to 100 years and remove the reference to the cosmic endowment.

Confidence: >80%. I'd like to put it higher but there's a lot of unknowns and I'm being conservative.

I'd be happy to bet money at the Caplan-Yudokowsky bet's 2-1 odds on a 13-year horizon. However, I don't think their clever solution actually works. The only way Yudkowsky benefits from his loan is if he spends all of his resources before 2030, but if he does that then Caplan won't get paid back, even if Caplan wins the bet. Morever, even Yudkowsky doesn't seem to treat this as a serious bet.

While there's no good way to bet that the world will end soon (except indirectly via financial derivatives), there is a way to bet that our basic financial infrastructure will continue to exist for decades to come: I am investing in a Roth IRA.

Reasoning

Biowarfare won't kill everyone. Nuclear war won't kill everyone. Anthropogenic global warming won't kill everyone. At worst, these will destroy civilization which, counterintuitively, makes Homo sapiens more likely to survive on the short term (century). The same goes for minor natural disasters like volcanic eruptions.

Natural disasters like giant meteors, or perhaps a gamma ray burst, are unlikely. The last time something like that happened was 66 million years ago. The odds of something similar happening in the next century are on the order of 2×10−6. That's small enough to ignore for the purposes of this bet. The only way everyone dies is via AI.

I see two realistic roads to superintelligent world optimizers.

Human simulator mesa optimizer running on a non-agentic superintelligence.

Inhuman world-optimizing agent.

Human simulators are unlikely to exterminate humanity by accident because the agent mesa optimizer is (more or less) human aligned and the underlying superintelligence (currently LLMs) is not a world optimizer. The underlying superintelligence won't kill everyone (intentionally or unintentionally), because it's not a world optimizer. The emulated human probably won't kill everyone by accident because that would require a massive power differential combined with malevolent intent. Another way humanity gets exterminated in this scenario is...
The Caplan-Yudkowsky End-of-the-World Bet Scheme Doesn't Actually Work
by lsusr — 2023-02-25T18:57:00.105Z

Caplan and Yudkowsky bet at 1-1 odds whether humanity would be exterminated from the surface of the Earth by 2030. How it works is Caplan pays Yudkowsky $100 in 2017 (when the bet began). If humanity is still living on the surface on the Earth in 2030 then Yudkowsky owes Caplan $200 plus 5% annual interest. See here for details.

I don't think Yudkowsky was taking this bet seriously, which somewhat invalidates it, but let's pretend the bet was in good faith. Does the bet actually work?

Some people criticized the bet on the grounds that it relied on Yudkowsky's honesty and interest rates staying similar. I don't think either of these are an issue. The bet contains a clause to deal with interest rates going haywire and there's no way even a psychopathic machiavellian Yudkowsky would trash his public reputation for a mere CPI-adjusted $200. Moreover, accounting for interest rates is something all time-displaced bets need to deal with. It's just an annoying fact of quantitative finance.

There's a bigger problem with the Caplan-Yudkowsky bet. The bet only makes sense from the perspective of Caplan. There's no direct way for Yudkowsky to intentionally profit from it except indirectly, via financial derivatives, unless our annihilation becomes certain.

Consider the most conservative approach. Yudkowsky invests his $100 in the stock market at 5% annual interest for 13 years. There are two possible outcomes: Either the world ends or it doesn't. If the world ends, Yudkowsky gains exactly nothing from the bet, because he never spent the money. If the world doesn't end then Yudkowsky simply loses money. If Yudkowsky is to profit from the bet, then he must spend money before the world ends in 2030.

Suppose it's 2029. Yudkowsky is confident that the world will end on January 1, 2030. There's no reason to preserve capital, so he blows all of his money on cocaine, prostitutes and a cryogenic vault. If the world ends, then great! But if the world doesn't end then Caplan doesn't get paid back. Yudkowsky can't spend the money as long as there is a chance the world will survive.

In other words, Yudkowsky can only benefit from the $100 by going bankrupt. (Unless interest rates change. Caplan notes that "this is primarily a bet that annualized real interest rates stay below 5.5%". Which is true. But what's interesting is the "End-of-the-World" part, not the 5.5% interest rate.)

It's pla...
The Practitioner's Path 2.0: the Empiricist Archetype
by  — 2023-02-25T17:05:12.689Z
Link: https://guildoftherose.org/articles/path-2-0-the-empiricist-archetype

Earlier this month, we introduced the Practitioner's Path 2.0, our new framework for structured self improvement. The heart of the Path is the Skilltree, which is divided into three archetypes: the Pragmatist, the Meditative, and the Empiricist.

The Practitioner's Path 2.0 launches on March 4th. Today, we'll go over the Empiricist archetype and her associated Skills.

The Empiricists see life as a puzzle, a complex system that can be manipulated into producing a desired outcome. Empiricists love to learn new facts and skills, even if they're not immediately useful. Theorizing about the world is fun too, but what the Empiricist really wants is to be right — and that means checking those clever hypotheses against the only judge that really matters: reality.

In the Empiricist tree, you'll find Skills focused on:

Synthesizing and distilling information
Learning and teaching
Thinking clearly

Some people might call Empiricists excessively logical, but the Empiricist knows that's a mischaracterization. Intuition is just as important as reason, and the Empiricist's ultimate goal is to practice thinking clearly so much that it becomes automatic.

Introductory Skill Examples

The Archetypal Empiricist

Eva Davis started her career as a nameless chemical technician with a slowly fading passion for science. Today, ten years later, she's a max-level Empiricist advancing the field of anti-aging medicine.

At work, Eva is a brilliant lab scientist for a biotech startup, Everlife. She devises experiments, runs them, and uses the results to, someday, allow the elderly to spend more time with their children and grandchildren. The reaper's scythe-hand weakens one day at a time.

Eva spends most of her time at work, but when she allows herself a rare moment of relaxation, Eva applies her razor-sharp mind to the world around her. Knowledge is everywhere — from the science papers she reads, to the natural history museum she frequents, to the endless stream of books she devours. She stays up late into most nights reading, telling herself that the exhaustion the next day is the price of knowledge.

The Practitioner's Path 2.0 launches March 4th. In the meantime, what do you think of the Empiricist archetype? Does it resonate with you? Let us know in the comments below!