trentmkelly/LessWrong-43k huggingface dataset:
SolidGoldMagikarp III: Glitch token archaeology
by mwatkins — 2023-02-14T10:17:51.495Z

The set of anomalous tokens which we found in mid-January are now being described as 'glitch tokens' and 'aberrant tokens' in online discussion, as well as (perhaps more playfully) 'forbidden tokens', 'unspeakable tokens' and 'cursed tokens'. We've mostly just called them 'weird tokens'.

GPT-3 speaks of 'the unspeakable one' when prompted about the enigmatic ‘ petertodd’

Research is ongoing, and a more serious research report will appear soon, but for now we thought it might be worth recording what is known about the origins of the various glitch tokens. Not why they glitch, but why these particular strings have ended up in the GPT-2/3/J token set.

['\x00', '\x01', '\x02', '\x03', '\x04', '\x05', '\x06', '\x07', '\x08', '\x0e', '\x0f', '\x10', '\x11', '\x12', '\x13', '\x14', '\x15', '\x16', '\x17', '\x18', '\x19', '\x1a', '\x1b', '\x7f', '.[', 'ÃÂÃÂ', 'ÃÂÃÂÃÂÃÂ', 'wcsstore', '\\.', ' practition', ' Dragonbound', ' guiActive', ' \u200b', '\\\\\\\\\\\\\\\\', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ', ' davidjl', '覚醒', '"]=>', ' --------', ' \u200e', 'ュ', 'ForgeModLoader', '天', ' 裏覚醒', 'PsyNetMessage', ' guiActiveUn', ' guiName', ' externalTo', ' unfocusedRange', ' guiActiveUnfocused', ' guiIcon', ' externalToEVA', ' externalToEVAOnly', 'reportprint', 'embedreportprint', 'cloneembedreportprint', 'rawdownload', 'rawdownloadcloneembedreportprint', 'SpaceEngineers', 'externalActionCode', 'к', '?????-?????-', 'ーン', 'cffff', 'MpServer', ' gmaxwell', 'cffffcc', ' "$:/', ' Smartstocks', '":[{"', '龍喚士', '":"","', ' attRot', "''.", ' Mechdragon', ' PsyNet', ' RandomRedditor', ' RandomRedditorWithNo', 'ertodd', ' sqor', ' istg', ' "\\', ' petertodd', 'StreamerBot', 'TPPStreamerBot', 'FactoryReloaded', ' partName', 'ヤ', '\\">', ' Skydragon', 'iHUD', 'catentry', 'ItemThumbnailImage', ' UCHIJ', ' SetFontSize', 'DeliveryDate', 'quickShip', 'quickShipAvailable', 'isSpecialOrderable', 'inventoryQuantity', 'channelAvailability', 'soType', 'soDeliveryDate', '龍契士', 'oreAndOnline', 'InstoreAndOnline', 'BuyableInstoreAndOnline', 'natureconservancy', 'assetsadobe', '\\-', 'Downloadha', 'Nitrome', ' TheNitrome', ' TheNitromeFan', 'GoldMagikarp', 'DragonMagazine', 'TextColor', ' srfN', ' largeDownload', ' srfAttach', 'EStreamFrame', 'ゼウス', ' SolidGoldMagikarp', 'ーティ', ' サーティ', ' サーティワン', ' Adinida', '":""},{"', 'ItemTracker', ' DevOnline', '@#&', 'EngineDebug', ' strutConn

...
The Linguistic Blind Spot of Value-Aligned Agency, Natural and Artificial

by Roman Leventov — 2023-02-14T06:57:58.036Z
Link: https://arxiv.org/abs/2207.00868

The paper by Travis LaCroix, July 2022.

Abstract:

The value-alignment problem for artificial intelligence (AI) asks how we can ensure that the 'values' (i.e., objective functions) of artificial systems are aligned with the values of humanity. In this paper, I argue that linguistic communication (natural language) is a necessary condition for robust value alignment. I discuss the consequences that the truth of this claim would have for research programmes that attempt to ensure value alignment for AI systems; or, more loftily, designing robustly beneficial or ethical artificial agents.

An attempt to sample the paper:

(1) Main Claim. Linguistic communication is necessary for (the possibility of) robust value alignment.
(2) Alternative Claim. Robust value alignment (between actors) is possible only if those actors can communicate linguistically.
(3) Contrapositive Claim. Without natural language, robust value alignment is not possible.

Thus, the simplest way of understanding why we should expect the main claim to hold is as follows:
(1) Principal-agent problems are primarily problems of informational asymmetries.
(2) Value-alignment problems are structurally equivalent to principal-agent problems.
(3) Therefore, value-alignment problems are primarily problems of informational asymmetries.
(4) Any problem that is primarily a problem of informational asymmetries requires information-transferring capacities to be solved, and the more complex (robust) the informational burden, the more complex (robust) the information-transferring capacity is required.
(5) Therefore, value-alignment problems require information-transferring capacities to be solved, and sufficiently complex (robust) value-alignment problems require robust information-transferring capacities to be solved.
(6) Linguistic communication is a uniquely robust information-transferring capacity.
(7) Therefore, linguistic communication is necessary for sufficiently complex (robust) value alignment.

If Bostrom (2014) is correct about language being an AI-complete problem, then linguistic AI is functionally equivalent to human-level AI. If the main claim is true, then linguistic AI is, in some sense, prior to robust value alignment. But, it was suggested above (2.2.3) that value alignment is logically prior to problems of control.68 The conjunction of these three insights leads to circularity, which may imply tha

...
Conceptual Pathfinding
by DirectedEvolution — 2023-02-14T05:49:51.856Z

Bootstrapping a company means starting a business with your own resources, without external funding. When learning from a textbook, we're not completely bootstrapping our understanding because we're not re-deriving the subject matter from scratch. While striving for conceptual understanding is common advice, bootstrapping the material by re-deriving it ourselves can be extreme.[1] An intermediate approach is "conceptual pathfinding," which uses textbook explanations but requires figuring out how to get from rough intuitions to detailed mathematical formalizations.

Conceptual pathfinding is a process that can help you build a deeper understanding of a concept. Here's how it works:

Start by stating the basic concept in simple, rough terms. Don't worry about being precise or mathematical, just try to capture the general idea.
Try to make your statement more specific, pushing yourself to see how far you can go. When you hit a dead end and can't see how to make your description more formal or solvable, move on to step 3.
Look up the description in a textbook to figure out the solution to your puzzlement. This can provide genuine insight and help you see how the formalization of the concept is a beautiful way to operationalize your intuition.
Clear your mind and repeat the process, starting from your rough intuition and working your way forward. Try not to draw on cached bits of information or intermediate results, and focus on figuring things out rather than straining to remember.
When you can get from your blank slate intuition to the final, detailed result, it should feel satisfying and like the concept has clicked into place. It's like moving to a new town and finally learning to navigate without a map.

By following these steps, you can deepen your understanding of a concept and build your intuition in a more organic and meaningful way.

As an engineering student, I find textbooks essential, but the prose descriptions often fall short. After skimming the text, I focus on hard theorems, proofs, equations, and diagrams to build a better understanding. However, this approach can make it difficult to retain information and re-derive math on my own. That's why I find it helpful to build intuition from the math and then use conceptual pathfinding to move from the core conceptual understanding back to the math.

For learning new concepts, I start with rough intuitions and ...
Important fact about how people evaluate sets of arguments
by Daniel Kokotajlo — 2023-02-14T05:27:58.409Z

Ronny Fernandez on twitter:

A very important fact which just came to my attention is that people do not tend to sum or take the max reasonableness of arguments for P to form a judgement about P, rather they tend to take the average.
This is a somewhat reasonable heuristic in some situations. For instance, if someone gives you a really unreasonable argument for P this is evidence that their judgement of arguments isn’t very good, and so their best argument is more likely to be secretly bad.
Similarly, it is evidence that they are motivated to convince you even using faulty arguments, which is generally speaking a bad sign.
It has important implications. Sometimes people think “oh I will make 50 ok arguments for P instead of one really good one” but most folks are not very impressed by this, even though they should be.
Relatedly, if you try to turn a complicated thesis T into a social movement, the average reasonableness of an argument in favor of T will plummet, and so you may very quickly find that everyone perceives the anti-T-ers as being much more reasonable.
This will probably still be true even if the best pro-T arguments are very good, and especially true if the best pro-T arguments are subtle or hard to follow.
Yes, this is about ai risk. I don’t think this is a slam dunk argument against trying to make ai-risk-pilled-ness into a popular social movement, but it is a real cost, and nearly captures the shape of my real worries.
The best version of my real worries, I also have real worries that are not nearly as defensible or cool.
Oh actually probably they take the min rather than the average unless they like you, in which case they take the max.

First of all, is this important fact actually true? I'd love to know. Reviewing my life experience... it sure seems true? At least true in many circumstances? I think I can think of lots of examples where this fact being true is a good explanation of what happened. If people have counterarguments or sources of skepticism I'd be very interested to hear in the comments.
Secondly, I concluded a while back that One Strong Argument Beats Many Weak Arguments, and in Ye Olden Days of Original Less Wrong when rationalists spent more time talking about rationality there was a whole series of posts arguing for the opposite claim (1, 2, 3). Seems possibly related. I'd love to see this debate revived, and tied in to the more ge...
How much is death a limit on knowledge accumulation?
by Gordon Seidoh Worley — 2023-02-14T03:54:16.070Z

The older I get, the more it feels like death is a major bottleneck to the accumulation of certain kinds of knowledge and insights.

It's not much of a blocker to shallow things that are easy to explain precisely. For example:

math
physics
accounting of money
stories (both fictional and historical)

It does seem to be a blocker for deep knowledge that's hard to easily explain and teach. For example:

rationality
philosophy
public policy & economics (in the broad sense of figuring out policies that organizations should implement to mould incentives in ways that achieve the desired outcomes)
how to live a good life

The former category of things benefit from being easy to put into writing and be transmitted accurately every time. This allows generations to easily build cumulative mountains of knowledge. Yes, each generation a person has to learn more before they can contribute to the mountain and, as of maybe 200 years ago, there's too much accumulation for one person to know everything known, but we don't seem especially close to the limit, and very recent advances like LLMs may offer a way to scale our learning further by outsourcing the remembering of many details.

The latter category suffers because each of them is hard to transmit and it's easy to be confused and transmit the wrong thing. Thus, to some extent, each person needs to rediscover the knowledge of past generations for themselves and only after the fact may recognize that previously generations had some of these same details figured out. Try as we might with institutions that teach young people the best of our wisdom of the past in these areas, it's imperfect and you have many people who will reject the knowledge because they don't understand it and will continue to reject it until they work it out for themselves. Thus even if somewhere there is an unbroken line from Socrates to the present day of ever advancing philosophical knowledge manifested in a single person, almost nobody would believe whatever they have to tell us because the inferential gap would be too large.

But is this story right? Can we quantify this? Is there some way we can measure how much knowledge we miss out on because we die and each generation has to spend time rediscovering things before they can advance it further? And how much additional knowledge would we accumulate, and how quickly, if death was not an obstacle?

I don't actually kn...
The Filan Cabinet Podcast with Oliver Habryka - Transcript
by MondSemmel — 2023-02-14T02:38:34.867Z

This is a transcript of The Filan Cabinet podcast, episode 6, with host Daniel Filan and guest Oliver Habryka. The transcript is by MondSemmel. The original LessWrong post is here.

In addition to each "section" having a timestamp, phrases/sentences that were difficult to understand (and where the transcript is therefore most likely to contain errors) will also be marked with a timestamp, if you would like to listen to those parts yourself.

Introduction [0:00:00]

Daniel Filan: Hello everybody. In this episode, I'll be speaking with Oliver Habryka. Oli is the head of the

Lightcone Infrastructure

team, the organization behind the forum

LessWrong

. He's also done grantmaking on the

Long-Term Future Fund

. Before we start, you should know that I've received funding from the Long-Term Future Fund. Oli was not involved directly in those funding decisions to fund me, basically, but he was also not recused from those decisions either. It's also the case that the Lightcone Infrastructure team runs offices in Berkeley that I am allowed to use for free.

With that out of the way, let's begin the interview. Well, Oli, welcome to the

Filan Cabinet

.

Oliver Habryka: Thank you.

Daniel Filan: So I understand that you lead the Lightcone Infrastructure team, which is... I think the project that people are probably most familiar with is running LessWrong, the website.

Oliver Habryka: Correct.

On LessWrong

The History of LessWrong [0:00:46]

Daniel Filan: So, do you want to tell me a bit about, like, there was an old version of LessWrong and then there was a new version of LessWrong and I guess you were involved in that. Can you tell me the story of the rejuvenation of the website?

Oliver Habryka: Yeah. I mean, very rough story: around 2008, Eliezer Yudkowsky started posting on

Overcoming Bias

, made a bunch of series of essays. After a while, he made so many essays that

Robin Hanson

, who was co-running the site with him, was like, “I don't know, man, how about you get your own apartment.”

So Eliezer moved out and invited a lot of friends with him and founded lesswrong.com. That site grew over a few years, first starting out when Eliezer was writing the

Sequences

, which is one of the most widely known series of essays that he's written, and then [he] started writing

Harry Potter and the Methods of Rationality

, which drove a lot more activity, which th...
Is InstructGPT Following Instructions in Other Languages Surprising? 
by DragonGod — 2023-02-13T23:26:28.594Z

On Twitter Jan Leike asks:

With the InstructGPT paper we found that our models generalized to follow instructions in non-English even though we almost exclusively trained on English.

We still don't know why.

I wish someone would figure this out.

I find myself surprised/confused at his apparent surprise/confusion.

My default response had someone asked me what I thought was going on would have been something like: "following instructions is a natural abstraction of a task", hence models trained to follow instructions in English generalising to other languages is a natural example of goals/capabilities generalisation.

It's like being surprised that you taught an AI to drive red cars and it can drive blue cars as well [capabilities generalisation]. Or if you taught an AI to reach a particular location on a level when there's a coin there and it still heads to said location in the absence of the coin. 🤭 [goal generalisation].

In Coin Run, there were multiple possible natural generalisations for the goal (location on level, the coin), but for instruction following in English, there seems to be only one natural generalisation of the goal? (I don't really see any other intuitively plausible generalisation of "following instructions in English" when the model switches to a Chinese context.)

Like conditional on InstructGPT competently responding in languages other than English, I think we should just expect it to generalise its goal to the new context.

Is this just hindsight bias? Would I really have been counterfactually surprised/confused if it didn't follow instructions in other languages? I don't actually know that to be the case.

Maybe I would have just generated some other reason why we shouldn't have expected instruction following to generalise.

But hindsight bias aside, this is a prediction of Wentworth's natural abstractions hypothesis as I understand it (I think he would endorse this prediction), and I am a genuine believer in the NAH, so I don't think it's just hindsight bias.
LLM Basics: Embedding Spaces - Transformer Token Vectors Are Not Points in Space
by NickyP — 2023-02-13T18:52:36.689Z

This post is written as an explanation of a misconception I had with transformer embedding when I was getting started. Thanks to Stephen Fowler for the discussion last August that made me realise the misconception, and others for helping me refine my explanation. Any mistakes are my own. Thanks to feedback by Stephen Fowler and JustisMills on this post.

TL;DR: While the token vectors are stored as n-dimensional vectors, thinking of them as points in vector space can be quite misleading. It is better to think of them as directions on a hypersphere, with a size component.

The I think of distance as the Euclidean distance, with formula:

d(→x1,→x2)=|→x1−→x2|=√∑i(x1i−x2i)2

Thus does not match up with the distance forumla used when calculating logits:

d(→x1,→x2)=→x1⋅→x2=|→x1||→x2|cosθ12

But it does match up with the cosine similarity forumula:

d(→x1,→x2)=^x1⋅^x2=cosθ12

And so, we can see that the direction and size matter, but not the distance

Introduction

In the study of transformers, it is often assumed that different tokens are embedded as points in a multi-dimensional space. While this concept is partially true, the space in which these tokens are embedded is not a traditional Euclidean space. This is because of the way probabilities of tokens are calculated, as well as how the behaviour of the softmax function affects how tokens are positioned in their space.

This post will have two parts. In the first part, I will briefly explain the relevant parts of the transformer, and in the second part, we will explore what is happening when a transformer moves from an input token to an output token explaining why tokens are better thought of as directions.

Part 1: The Process of a Transformer

Here I will briefly describe how the relevant parts of the transformer work.

First, let's briefly explain the relevant parts at the start of the transformer. We will be studying the "causal" transformer model (ie: that given N tokens, we want to predict the (N+1)th token). The main "pieces" of a causal model are:

The Tokeniser - turns "words" into "tokens" or "tokens" into "words"
The Transformer - turn N tokens into a prediction for the (N+1)th token

Input Embedding - turns "tokens" into "vectors"
Positional Encoder - adds information about position of each token
Many Decoder layers - turns input vectors into prediction vectors

with a Self-Attention sub-layer - uses information from all st

...
4 ways to think about democratizing AI [GovAI Linkpost]
by Orpheus16 — 2023-02-13T18:06:41.208Z
Link: https://www.governance.ai/post/what-do-we-mean-when-we-talk-about-ai-democratisation

Many AI labs have called for the democratization of AI. In a recent GovAI blog post,

Elizabeth Seger

summarizes four different ways of interpreting the phrase:

Democratizing AI use: Making it easier for people to use AI technologies
Democratizing AI development: Making it easier for a wider range of people to contribute to the development and design of AI systems
Democratizing AI benefits: Ensuring that the benefits of AI systems are widespread
Democratizing AI governance: Ensuring that decisions involving AI systems are informed by a variety of stakeholders and reflect democratic processes

Things I like about the post

“Democratizing AI” is a vague phrase, and the post usefully distinguishes between various ideas that the term can refer to.
The framework can help us distinguish between forms of democratization that are relatively safe and those that carry more risks.

Ex: Democratizing AI benefits seems robustly good, whereas democratizing AI use has risks (given that AI can be

dual-use

).

The framework could allow AI developers to maintain their commitment to (some forms of) democratizing AI while acknowledging that some forms carry risks.
The post analyzes decisions by AI labs through the lens of the framework. Example:

In declaring the company’s AI models will be made open source, Stability AI created a situation in which a single tech company made a major AI governance decision: the decision that a dual-use AI system should be made freely accessible to all. (Stable diffusion is considered a “dual-use technology” because it has both beneficial and damaging applications. It can be used to create beautiful art or easily modified, for instance, to create fake and damaging images of real people.) It is not clear, in the end, that Stability AI’s decision to open source was actually a step forward for the democratisation of AI governance.

The post is short! (About 5 minutes)

Read the full post

here

.
Does the AGPL Work?
by jefftk — 2023-02-13T14:20:01.674Z

It might be important to you that users of software you write are able
to modify it, in which case you might release it with a

copyleft license
like the

GPL. For
example, imagine someone's making a device that will run Linux. They
need to tweak Linux so it will work on this particular device, and
they do that and sell you a copy. Because Linux is under the GPL,
however, they also need to publish the source code for their tweaks.

Then along comes the web, and people no longer send out copies of their
programs. Instead, the programs live on their server, and you talk to
them with your browser:

This means that even if a program is under the GPL, the people
interacting with it have no right to the source code. This was called
the "application service provider loophole", and the AGPL was
created to fix it. Mastodon
is an example of AGPL software, and if you visit an instance you'll
see a little "view source code" link. This isn't just something your
instance is doing to be nice—it's required by the AGPL.

Or is it? A few months ago I read an interesting
argument that you can easily bypass the AGPL: stick a "proxy"
between the AGPL code and the user, which removes the source code
offer:

How is this compliant with the license? The AGPL tells you that when
you modify the program you must preserve its "offer the source to
anyone who asks" behavior:

if you modify the Program, your modified version must prominently
offer all users interacting with it remotely through a computer
network (if your version supports such interaction) an opportunity to
receive the Corresponding Source of your version by providing access
to the Corresponding Source from a network server at no charge,
through some standard or customary means of facilitating copying of
software.

The modified software still offers everyone the option to download the
source, but then a different piece of software that isn't covered by
the license runs a sub_filter 'source_url' '' and the
user doesn't actually receive the offer.

Courts do care about intent and this is, of course, not what the AGPL
authors intended, but it's not clear to me that what they intended is
coherent? They mechanism they chose, a license that controls what
you're allowed to do when making changes to the AGPL-covered software,
doesn't seem like it would be able to prevent someone from...