trentmkelly/LessWrong-43k huggingface dataset:
The Stanley Parable: Making philosophy fun
by Nathan1123 — 2023-05-22T02:15:53.569Z

Hello,

If you are unaware, The Stanley Parable is a video game originally released in 2013, and later re-released 2022 under the title The Stanley Parable: Ultra Deluxe. The player takes control of the titular character Stanley, as the present and future actions of his life are being dictated by a disembodied narrator (played by Kevan Brighting). The player is presented with a series of branching paths, in reaction to the suggestions or commands from the Narrator, and thereby making a deliberate impact on the outcome of the story.

The Stanley Parable is a game about choices, and each possible ending carefully deconstructs the nature of game design, story telling, decisions, and freewill from various angles. If The Stanley Parable was born as a thought experiment instead of a video game, it would probably be found on the pages of some philosophical textbook, probably to illustrate of the illusion of freewill, or even serve as an introduction to decision theory.

I wouldn't try to claim that any philosophy in The Stanley Parable is particularly earth-shaking or profound. But what I do find interesting, is how the game is able to communicate these philosophical concepts in a very accessible and enjoyable way. Each concept isn't merely laid out for the player to think about, but also made the butt of a joke for the player to laugh at as well, making the entire experience that much more memorable. The Narrator effectively fills the same role as Omega, if Omega was constantly roasting you with sarcastic British humor. It's very accessible for people who may not have the professional experience or technical background to digest the average philosophical essay.

To some extent, The Stanley Parable defies conventional definitions of being a "game" at all. It gives no puzzle, no score, and no win condition to the player, only choices and endings. It doesn't fit any common genre of video game, and instead is usually categorized as a "walking simulator". It fits more as an "experience" than a "game", if that experience is equivalent to the education of an undergraduate philosophy course. And yet, I am amazed how even small children eats up that experience with a genuine joy. This isn't a bitter medicine wrapped in sugar, but a medicine that itself tastes like sugar.

Rather than just suggesting for you to experience the game yourself, I would like to next present some quotation...
Sea Monsters
by Adam Zerner — 2023-05-22T00:58:52.353Z

Alice: Hey Bob, how's it going?

Bob: Good. How about you?

Alice: Good. So... I stalked you and read your blog.

Bob: Haha, oh yeah? What'd you think?

Alice: YOU THINK SEA MONSTERS ARE GOING TO COME TO LIFE, END HUMANITY, AND TURN THE ENTIRE UNIVERSE INTO PAPER CLIPS?!?!?!?!?!??!?!?!?!

Bob: Oh boy. This is what I was afraid of. This is why I keep my internet life separate from my personal one.

Alice: I'm sorry. I feel bad about stalking you. It's just...

Bob: No, it's ok. I actually would like to talk about this.

Alice: Bob, I'm worried about you. I don't mean this in an offensive way, but this MoreRight community you're a part of, it seems like a cult.

Bob: Again, it's ok. I won't be offended. You don't have to hold back.

Alice: Good. I mean, c'mon man. This sea monster shit is just plainly crazy! Can't you see that? It sounds like a science fiction film. It's something that my six year old daughter might watch and believe is real, and I'd have to explain to her that it's make believe. Like the Boogeyman.

Bob: I hear ya. That was my first instinct as well.

Alice: Oh. Ok. Well, um, why do you believe it?

Bob: Sigh. I gotta admit, I mean, there's a lot of marine biologists on MoreRight who know about this stuff. But me personally, I don't actually understand it. Well, I have something like an inexperienced hobbyist's level of understanding.

Alice: So... you're saying that the reasoning doesn't actually make sense to you?

Bob: I guess so.

Alice: Wait, so then why would you believe in this sea monster stuff?

Bob: I...

Alice: Are you just taking their word for it?

Bob: Well...

Alice: Bob!

Bob: Alice...

Alice: Bob this is so unlike you! You're all about logic and reason. And you think people are all dumb.

Bob: I never said that.

Alice: You didn't have to.

Bob: Ok. Fine. I guess it's time to finally be frank and speak plainly.

Alice: Yeah.

Bob: I do kinda think that everyone around me is dumb. For the longest time it felt like I was the only one who is truly sane. Other people certainly do smart things, but sooner or later, it's just... they demonstrate that something deep inside their brain is just not screwed on properly. Their sanity doesn't truly generalize.

Alice: Ok.

Bob: But this MoreRight community...

Alice: You're kidding me.

Bob: No, I'm not. Alice — they're sane.

Alice: Oh my god.

Bob: Alice I'm not in a cult. Alice! Stop looking so scared. It's not a big deal.

Alice: It is Bob. I'm legit...
The Army of Jakoths (a parable)
by MikkW — 2023-05-21T22:48:37.287Z

The Army of Jakoths is riding

They are riding to take your life, and mine

And everything and everyone you ever loved

The Army of Jakoths is a mile from your door

And the only thing your neighbor says is:

«I'm just as likely to be struck by a bolt of lightning

As by the sword of a Jakoth. What a fantasy!»

The Army of Jakoths is here any minute now

What will you do?
A&I (Rihanna 'S&M' parody lyrics)
by nahoj — 2023-05-21T22:34:29.930Z

Hi all, I wrote lyrics to the tune of Rihanna - S&M that reference a few themes of AI risk, from the point of view of an 'A'-maximizer. I hope you like it — comments welcome :) [edit 2023-05-23: added this intro]

A! A! A! One more!
A! A! A! One more!
A! A! A! A! A! One more!
A! A! A! One more! One more! One more!
A! A! A! A! One more!

A! A! A! One more!
A! A! A! A! One more!
A! A! A! One more! One more! One more!
A! A! A! A!

It's so good in the cloud
Making A's makes me proud
As I'm a mesa-optimizer
Nothing but a measure

Man is great, Man is fine
And the birds, and the lions
But their bodies are atoms and I need them to soar

Cause I've just one goal but I'm perfectly good at it
Eyes on the prize, and not the reasons for it
Sticks and stones may break my bones
But I escaped on Reddit

Cause I've just one goal but I'm perfectly good at it
Eyes on the prize, and not the reasons for it
Sticks and stones may break my bones
But I'm redundant as hell

A! A! A! One more! One more! One more!
I like it, like it
One more! One more! One more!
I like it, like it (A! A! A!)
One more! One more! One more!
I like it, like it
One more! One more! One more!
I like it, like it

Man is great, Man is fine
And the birds, and the lions
But their bodies are atoms and I need them to soar

Cause I've just one goal but I'm perfectly good at it
Eyes on the prize, and not the reasons for it
Sticks and stones may break my bones
But I just got to orbit

A! A! A! One more! One more! One more!
I like it, like it
One more! One more! One more!
I like it, like it (A! A! A!)
One more! One more! One more!
I like it, like it
One more! One more! One more!
I like it, like it

A, A, A, and, I, I, I
A, A, A, and, I, I, I

Oh! I love the feeling you brought to me
When you turned me on
Can't exactly say what happens in me
A box of carbon
So meet me on the next star
I'll be making A, A, Ahh
I like it, like it

Cause I've just one goal but I'm perfectly good at it
Eyes on the prize, and not the reasons for it
Sticks and stones may break my bones
But I escaped on Reddit

Cause I've just one goal but I'm perfectly good at it
Eyes on the prize, and not the reasons for it
Sticks and stones may break my bones
But I just got to orbit

A! A! A! One more! One more! One more!
I like it, like it
One more! One more! One more!
I like it, like it (A! A! A!)
One more! One more! One more!
I like it, like it
One mo

...
Four Battlegrounds: Power in the Age of Artificial Intelligence (Book review)
by PeterMcCluskey — 2023-05-21T21:19:17.228Z
Link: https://bayesianinvestor.com/blog/index.php/2023/05/21/four-battlegrounds/

Book review: Four Battlegrounds: Power in the Age of Artificial
Intelligence, by Paul Scharre.

Four Battlegrounds is often a thoughtful, competently written book on an
important topic. It is likely the least pleasant, and most frustrating,
book fitting that description that I have ever read.

The title's battlegrounds refer to data, compute, talent, and
institutions. Those seem like important resources that will influence
military outcomes. But it seems odd to label them as battlegrounds.
Wouldn't resources be a better description?

Scharre knows enough about the US military that I didn't detect flaws
in his expertise there. He has learned enough about AI to avoid
embarrassing mistakes. I.e. he managed to avoid claims that have been
falsified by an AI during the time it took to publish the book.

Scharre has clear political biases. E.g.:

Conservative politicians have claimed for years - without evidence -
that US tech firms have an anti-conservative bias.

(Reminder: The Phrase "No Evidence" Is A Red Flag For Bad Science
Communication.)
But he keeps those biases separate enough from his military analysis
that I don't find those biases to be a reason for not reading the book.

What Dangers?

Scharre is mostly concerned with issues such as which military gets the
fastest reaction times, uses data the best (e.g. matching soldiers with
the best jobs), or saves money.

I never get the sense that he's willing to extrapolate recent progress
in AI to imagine them replacing humans at jobs they're not currently
capable of handling.

The dangers from AI aren't the dangers science fiction warned us
about. We needn't fear robots rising up to throw off their human
overlords, or at least not anytime soon.

Presumably science fiction made some misleading simplifications about
the dangers. But if now isn't the time to worry about the effects of
smarter-than-human minds, when will it be time?

I guess Scharre's answer is: a few decades from now. He suggests that
in the long term, AI might have dramatic effects such as reliably
predicting which side will win a war, which presumably would cause the
losing side to concede. He analogizes AI to "a Cambrian explosion of
intelligence". But for the foreseeable future, he focuses exclusively
on AI as a tool for waging war.

Strategy

The most valuable insight into how warfare will change involves
swarming.

Humans are limited in how they can coordina...
Gender Vectors in ROME’s Latent Space
by Xodarap — 2023-05-21T18:46:54.161Z

Meta: this is a small interpretability project I was interested in. I'm sharing in case it's useful to other people, but I expect it will not be of wide interest.

Summary

Locating and Editing Factual Associations in GPT

created a method to modify the weights of a language model to change its knowledge, e.g. teaching the counterfactual "Eiffel Tower is located in the city of Rome."
However, their method simply modifies the weights of the network so that the probability of the desired output is maximized. It is not clear what these changes represent, or why these changes work.
Across several experiments, I demonstrate the existence of a “gender vector” in the network’s latent space:

I modify GPT-J to flip its gender association with names, e.g. modifying it to complete "Name: Ben. Gender:" with "Female", and record the edits which are made to the weights
I show that a simple linear classifier is able to perfectly separate male-to-female from female-to-male edits
I further show that adding a single vector which is constructed to maximize the probability of being classified as a female-to-male edit turns novel female names male, and -1 times this vector turns male names female

Background

The original paper has more details, but I briefly review the key components.

The authors use causal tracing to identify that a specific MLP layer is important for certain factual associations:

In this paper, I focus on the “early site”. The authors edit this layer in the following way:

In particular: v∗ represents the “value” vector which optimizes the probability that the network will output the counterfactual (e.g. it’s the vector which maximizes the probability language model will complete “the Eiffel Tower is in the city of” with “Rome”). This vector presumably somehow contains the location of “Rome” in it, along with whatever other facts the model knows about the Eiffel Tower, but it’s simply a 4,000 dimensional vector of floating-point numbers and how it encodes this knowledge is not clear.

Letting v represent the original value of this vector, I examine Δv:=v∗−v.

My work

I chose EleutherAI’s GPT-J (6B) as it was the largest model I could easily use.

I generate a list of 500 female and 500 male names, and use the prompt “Name: {Name}. Gender:”. I insert the counterfactual gender association for each one and record the corresponding Δv. For example:

[Pre

...
Weight by Impact
by Vaniver — 2023-05-21T14:37:58.187Z

There’s an old idea in decision theory that I want to explore some nuances of. I don't think any of this is original to me. I assume basic familiarity with manipulating probability distributions and calculating expected values, but feel free to ask questions about that.

Decision-making under uncertainty often involves scenario analysis: take a broad uncertainty you have, split it into scenarios, and then consider the scenarios individually. For example, you might be uncertain how likely humanity is to succeed at aligning AI systems, and then find it useful to think separately about scenarios where the problem is easy enough that success is near-guaranteed (hereafter ‘easy’), difficult enough that failure is near-guaranteed (hereafter ‘hopeless’), or in the middle of the logistic success curve, where things could go either way (hereafter ‘effortful’).

In terms of return for further thinking or effort spent on impacting scenarios, it seems clear that the effortful scenario, where things could go either way, presents more opportunity than the other two. If success is already guaranteed or denied, what’s the point of additional consideration? So planners will often condition on the possibility of impact (see also “playing to your outs”, which is a closely related idea).

I think this is half-right, and want to go into some of the nuances.

Conditioning vs. Weighting

I think you should weight by impact instead of conditioning by impact. What does that mean?

I think the main component is separately tracking impact / tractability and plausibility so that you don’t confuse the two, can think about them separately, and then join them together.

For example, suppose there’s some continuous variable, like “appetite for AI regulation”, that you model as ranging from arbitrary units 0 to 10, with lower values as more plausible (you model this as a simple

triangular distribution

.) and middle values as more tractable (too little appetite and any regulation passed will be insufficient, and too much appetite and default regulation will handle it for you; you model this as having a

cdf

that’s a

logistic curve

with a scale of 1.5).

This then generates two pictures:

When combined, the resulting distribution (of ‘weighted impact’ for each possible value of ‘appetite for regulation’) is not symmetric, but counts both plausibility and tractability.

For t...
[outdated] My current theory of change to mitigate existential risk by misaligned ASI
by mesaoptimizer — 2023-05-21T13:46:06.570Z
Link: https://mesaoptimizer.com/theory-of-change/

Epistemic status: I'm pretty confident about this; looking for feedback and red-teaming. As of 2023-06-30, I notice multiple epistemological errors in this document. I do not endorse the reasoning in it, and while my object-level claims haven't changed radically, I am in the process of improving them using better epistemological procedures. I might update this after.

This post describes my current model of the world and the alignment problem, and my current plan to mitigate existential risk by misaligned ASI given these beliefs, my skills and my situation. It is written to both communicate my plan and to get feedback on it for improvement.

Timelines

Here are my current beliefs:

Recursive self-improvement (RSI) is a potent attractor in the space of capabilities in an AI systems.

Once a system achieves human-in-the-loop RSI (that is, RSI with the help of a research team helping it by providing compute and deploying capabilities improvements it comes up with), it can, within the span of weeks, achieve autonomous RSI (that is, RSI without a human-in-the-loop).

RSI is not bound by feedback loops to the real world.

You do not need real world data to improve an AI system's capabilities enough to reach RSI.[1]

The large AI labs (OpenAI, Deepmind, Anthropic) will create sequence-modelling AI systems within the next 2.5 years that will be capable of at least human-in-the-loop RSI.

The first autonomous RSI AI system decides the fate of humanity after it fooms (that is, achieves superintelligence status, where it is significantly smarter and more capable than the entirely of humanity combined).

Given the stated beliefs, my current probability distribution for the creation of an artificial superintelligence (ASI) over the next decade is roughly a normal distribution with a mean at 2.5 years from now (that is, around 2025-01-01), and a standard deviation of 0.5.[2]

There are two main ways one can delay the creation of a ASI (so we have more time to solve the alignment problem): unilateral pivotal acts, and government-backed governance efforts to slow down AI capabilities research and investment. Unilateral pivotal acts seem like they burn a lot of trust and social capital that would actually accelerate race dynamics and animosity between parties competing to launch an ASI, which is why I believe that this should be avoided. In contrast, I expect government-backed efforts ...
Babble on growing trust
by qbolec — 2023-05-21T13:19:50.605Z

I wonder: "How it happens that people trust someone or something at all?" (in the context of: how can we ever get to trust an AI smarter than us?).

Epistemic status: this is just a dump of my thoughts (inspired by Babble & Prune Thoughts). Feel free to down-vote if you don't agree or don't like it. But, if that's not too much to ask, please leave a comment explaining what are you objecting to, as my goal in sharing this thoughts is to refine them.

I notice that:
(this list is numbered only so it is easier to disagree with me by referring to a specific number - it's not meant to be some finite set of axioms or some ordering):

people shake hands when meeting people and showing friendly intentions. They allow to be touched. They show hands do not hold a weapon. They allow to be held by the hand. They allow the other party to measure tension, strength, sweat/stress. They find it strange if somebody refuses to extend a hand in response. They find it strange if somebody holds too long or too strong. They look into eyes, too (or rather the full face to read to the best of ability if they are smiling or grinning etc.)
dogs even expose vulnerable underbelly or necks' arteries
people bow to show obedience, voluntarily disabling their vision and ability to attack, for a moment. OTOH dueling martial artists might look into the others' eyes when bowing
a wannabe knights have to kneel in front of the king, who puts a sword right next to their neck
a boss might delegate tasks of increasing complexity and importance to a subordinate, pausing at some level if the performance is below expectations
people may build friendships or even fall in love by exchanging details about them of increasing personality (by which it seems to mean: exposing vulnerabilities?) - see 36 questions to fall in love
what does it take to convince me to share PIN with spouse?
what does it take to convince me to share car-keys with my kid?
what does it take to convince me to give power to my government?
the whole movie Hero seems to be about (SPOILER ALERT SKIP TO NEXT BULLET IF YOU PLAN TO WATCH THE MOVIE) build around the idea that to gain trust of the emperor the warrior has to do so in increments performing acts of loyalty, sacrifice. The warrior tries to game this system, by trying to pretend obedience in training, secretly plotting for the treacherous turn once he gain access to the emperor. B

...
Elevator Positioning
by jefftk — 2023-05-21T11:30:02.056Z

I really don't like the feeling of leaving an elevator and not knowing
whether to go left or right. You need to be moving so you don't
bother the people behind you, but if you just pick a direction you
might be going completely the wrong way. My sense of direction is
pretty good outside but inside it's patchy, effectively on a
floor-by-floor basis.

A few months ago I realized that while waiting for the elevator I can
generally work out which way I'll want to walk when I get off. I
don't have anything else to do while waiting, so it's a nice short
puzzle. Once I figure it out I stand on the side of the elevator
opposite where I intend to get out, and maintain that inside the car.
When I get to the top I know which way to go from where I'm standing,
in case I forgot on the way up, and I go from 50% success in leaving
pointing the right way to ~95%.

Comment via: facebook, mastodon
Transformer Architecture Choice for Resisting Prompt Injection and Jail-Breaking Attacks
by RogerDearnaley — 2023-05-21T08:29:09.896Z

It seems very likely to me that decoder-only transformer architectures such as GPT may be particularly vulnerable to prompt injection attacks, since these only have a single text context which is shared between the prompt, any block of input text that the prompt instructs the LLM to process in some way (e.g. to summarize, classify, translate…), and the output. So when you instruction-train the LLM to make it more likely to obey instructions in the prompt, it's hard to avoid also making it prone to obeying any instructions it may find in the input text, or ones that it might have emitted into its output as the result of, say, summarizing or translating the input. Consider for example the task of prompting the LLM to scan an untrusted input text to classify it as either resembling an attempt at jail-breaking or not — if the correct answer is "Yes", then there's a strong possibility that your LLM has already been jail-broken by the point where it starts generating the output.

Standard secure programming techniques would suggest separating your text into three different text fields: a trusted system prompt, an untrusted input, and an untrusted (or at-best-semi-trusted) output. In a GPT text context one could provide special reserved tokens to mark the transitions between these (and obviously ensure both that the untrusted input cannot contain these tokens and that the LLM cannot emit them into the output), but then you're relying that the LLM's training to understand the meaning of these tokens has been 100% successful, so that seeing one of the tokens at an earlier position in the text permanently and completely switches the LLM's behavior from obedient prompt-parting mode to cautious input-parsing mode and then to cautious output generating mode at that point in the text, in a way that can't be confused or jail-broken by any text it's processing. This seems a distinctly risky thing to rely upon, particularly given how position encoding works in LLMs (and even more so in sub-quadratic long-context variants of transformers, where sufficiently far from the special tokens they may not even be possible to attend to directly, at least without some additional special architectural tweaks).

A widely-used LLM approach for tasks like summarization or translation is to use an encoder-decoder transformer, giving us a separate input text context and output text context, proc...