trentmkelly/LessWrong-43k huggingface dataset:
Clarifying wireheading terminology
by leogao — 2022-11-24T04:53:23.925Z

See also: Towards deconfusing wireheading and reward maximization, Everett et al. (2019).

There are a few subtly different things that people call "wireheading". This post is intended to be a quick reference for explaining my views on the difference between these things. I think these distinctions are sometimes worth drawing to reduce confusion.

Specification gaming / reward hacking: The agent configures the world in such a way that makes its reward(/utility) function achieve a high value in a way that is not what the creators intended the reward function to incentivize.

Examples: the boat race example, our recent work on reward model overoptimization

Reward function input tampering: The agent tampers with the inputs to its reward function to make the reward function see the same inputs as it would when observing an actual desirable worldstate.

Examples: sensor tampering, VR

Reward function tampering: The agent changes its reward function.[1]

Examples: meditation, this baba-is-you-like gridworld

Wireheading: The agent directly tampers with the reward signal going into the RL algorithm.

Examples: dopamine agonists, setting the reward register

Some crucial differences between these:

1 is a problem of both embedded[2] and non-embedded settings. 2 is a problem of partially-observedness[3], which is unavoidable in embedded settings, but also comes up in many non-embedded settings as well. 3 and 4 are problems of embeddedness; they do not show up in non-embedded settings.
2, 3, 4 are all about "not caring about the real world" in various ways, whereas 1 is about "caring about the real world in the wrong way"
4 is about as close as you can get to a "reward-maximizing policy" in an embedded setting.

^

Note: this is not the same thing as changing a terminal goal! The reward function is not necessarily the terminal goal of the policy, because of inner misalignment.

^

Here, by "embeddedness" I mean in the sense of Demski and Garrabrandt (2019); in particular, the fact that the agent is part of the environment, and not in a separate part of the universe that only interacts through well defined observation/action/reward channels. RL is the prototypical example of a non-embedded agency algorithm.

^

That is, the reward function is unable to perfectly observe the ground truth state of the environment, which means that if there is another state the world cold be in that

...
LW Beta Feature: Side-Comments
by jimrandomh — 2022-11-24T01:55:31.578Z

LessWrong now has side-comments. This feature is in beta; you can turn it on for yourself on individual posts using the triple-dot menu below the post title, or enable it for all posts by going to your user settings and checking the "Opt into experimental features" checkbox in the Site Customization section.

Side-coments on LessWrong are conceptually similar to the side-comments you may be familiar with from Google Docs and other places, with one key difference: side-comments are placed automatically by lining up blockquotes. As a result, many historical posts already have side-comments on them! Side-comments are also still displayed in the comments section below the post as usual.

Side-comments take the form of a comment icon in the right margin, which expands when you mouse over it to show a comment. Click on the icon to pin it open. It looks like this:

To create a side-comment on a post, just write a normal comment which quotes an excerpt from the post, in blockquote format. (To create a blockquote, type > ). You can also highlight some text and click the button that appears to start a comment pre-seeded with a quote containing that text.

By default, side-comments are filtered to comments with 10+ karma (plus comments by the post author). They aren't available on mobile and require that your screen is wide enough to have some space in the right margin.

If a comment contains more than one blockquote that could be used to place it as a side-comment, it will be placed based on the first one. In order to appear as a side-comment, quotes should be an exact match, including formatting; there is some provision for "..." ellipses, but it's fairly limited.

Please give feedback on how this affects the reading experience! If the feedback is positive, this will leave beta and be enabled (for comments above a karma threshold) for everyone.
Against "Classic Style"
by Cleo Nardo — 2022-11-23T22:10:50.422Z

What is Classic Style?

The Sense of Style is Steven Pinker's style guide informed by cognitive psychology and linguistics. The main idea is that the author should write in a particular mode of communication called "classic style".

The guiding metaphor of classic style is seeing the world. The writer can see something that the reader has not yet noticed, and he orients the reader’s gaze so that she can see it for herself. The purpose of writing is presentation, and its motive is disinterested truth. It succeeds when it aligns with the truth, the proof of success being clarity and simplicity.

— Clear and Simple as the Truth, Francis-Noel Thomas and Mark Turner

Here are the do's and don't of classic style:

Eliminate meta-discourse.
Don't write "in the next section I will..."
Eliminate "hedging".
Don't write "maybe", "it seems to me", or "I think that".
Don't use concepts about concepts.
Don't write "approach", "assumption", "concept", "condition", "context", "framework", "issue", "level", "model", "paradigm", "perspective", "process", "role", "strategy", "tendency", "variable".
Talk about the subject, not about research about the subject.
If you're writing about apples, then write "apples are X" rather than "Dr Smith first discovered that apples are X".

Roughly speaking, Pinker recommends that the author should write about something as if neither the document nor the author nor the reader actually exists. If they're writing a book about apples, then the sentences should assert facts about apples, not about the study of apples and certainly not about the book itself.

My opinion: I like that Pinker deduces his advice from an underlying theory of communication. This is better than many style guides which present an ad-hoc list of tips. However, I think the theory is wrong. [1]

What's wrong with Classic Style?

I'm suspicious of "A Sense of Style" because it separates what you should write and what is true. To clarify, Pinker isn't saying there are things you should write that aren't true, but he is saying there are things that are true that you shouldn't write.

Pinker contrasts "classic style" with what he calls "postmodern style" — where the author explicitly refers to the document itself, the readers, the authors, any uncertainties, controversies, errors, etc. I think a less pejorative name for "postmodern style" would be "self-aware style". This is the predo...
Meme Dialects
by jefftk — 2022-11-23T21:30:05.008Z

When I started at Google about ten years ago I was surprised to learn
that image macros ("memes") were very popular internally. There was a
page for creating, sharing, voting on, and discussing them, "Memegen"
[1], and it was rare for someone to give a presentation without
including some. My initial reaction was pretty negative: my earlier
exposure to memes has been low-quality jokes in places where they
tended to crowd out better discussion. In this context, however, they
weren't actually crowding other things out, and instead Memegen's
listing for highly-upvoted memes provided one of the best overviews of
company-internal news.

Memes often make heavy use of shared context. Initially this was
broader culture, but well-known previous memes feed back into cultural
references available to draw on, and you get the beginnings of a
language. For example, an image of a guy with an ugly
brown cap became a standard way of talking about a particular kind
of selfish or unreliable behavior. Each meme using that image
reinforces and expands the association, increasing how much you can
communicate by using it as the background for another meme. The hat
is recognizable enough on its own that people will draw on those
associations by pasting the hat in other contexts.

With tens of thousands of people creating and sharing Google-internal
memes for over a decade, the internal associations from past memes
diverged from external ones. Not entirely, since people weren't cut
off from the outside world, but in a way much like a dialect. As
someone who's read ~hundreds more Google-dialect memes than
general-internet ones, this is something I need to think about when I
(as I very occasionally do) make memes.

For example, remembering an old 80,000 Hours article
and thinking about the FTX collapse I put
something together for the EA memes FB group:

I selected and adapted the image of Bankman-Fried at the bottom to
resemble the "challenge
accepted" stick figure that is standard in the general internet
dialect:

This felt a bit like speaking a foreign language, however, since (per
memory, can't check anymore) inside Google people would normally use
variants of this image of someone drinking beer:

Until yesterday, however, I didn't know that on the broader internet
that image has associations of burned-out
college students. I'm glad I realized ...
When do you visualize (or not) while doing math?
by Alex_Altair — 2022-11-23T20:15:20.885Z

I've been doing a lot more math lately, and, of course, also thinking about how I can do it faster and with deeper understanding. When I was in college, I remember that I found visualizing immensely helpful, to the point where it's almost what "doing math" is for me.

The most obviously useful place to visualize was in multivariable and vector calculus. Especially because so many things were set in three dimensions, visualizing expressions made it so much easier to see why they were true.

I also visualize all the way down to doing algebra. When I symbolically manipulate an equation, I visualize things like bringing the letters across, or them merging into each other.

But it has its downsides. One is that, since I'm visualizing, I'm visualizing something specific; a specific function, a specific type of set. It's not necessarily high-resolution, but it don't usually cover the full range of what is represented by the symbols on the paper. And so by visualizing something specific, it's likely that I'm not keeping in mind all the types of cases.

The most obvious downside is that visualization is computationally intensive. When I'm rendering a big scene in my mind, I can feel that it's hard. Sometimes I have to close my eyes and block out the light. Things can slow down, and it can take several tries to get the visual just right, or to maintain it all the way through the manipulation. It seems plausible that this gets me more tired than is necessary, and causes me to stop doing the task earlier.

So, it has at least occurred to me that I could "try" doing something else. I put try it quotes because visualizing is so natural that it would be effortful for me to even think of an alternative. It's as if someone said "you know, you don't have to shovel that snow by hand" and I'm thinking, do they mean I could do it with my feet? What they actually mean is using a snow-blower.

Are there fundamentally different ways that you do math? Are there particular tasks you've tried that go faster with one method or another? Are there books that talk about this type of strategy?
When AI solves a game, focus on the game's mechanics, not its theme.
by Cleo Nardo — 2022-11-23T19:16:07.333Z

A game design consists of two things: mechanics and theme.

The game mechanics is the abstract protocol governing how the players interact with each other and their shared environment.
The game theme is a fictional interpretation of the elements of the game.
Consider Battleship — the theme is a naval battle, and the mechanics are a particular 2-player sequential discovery game.
There is a correspondence between the ontology of the mechanics and the ontology of the theme, but this correspondence is mostly arbitrary. For example, the Knight in chess has almost nothing to do with actual knights.

When an AI solves a game, people often overfocus on the theme of the game relative to the mechanics of the game.
Maybe this is for psychological reasons:

The theme is more interesting than the mechanics.
The theme is in our pre-cached ontology — my brain already has a pre-cached concept of "naval battle" but it doesn't have a pre-cached concept corresponding to the particular mechanics of Battleship. In fact, this is why games have themes in the first place — they serve partly as mnemonics for the rules.
Did you know that people cooperate more in the Prisoner's Dilemma if the game is called "The Community Game" than "The Wall Street Game"?!

Or maybe this is for rational reasons:

Other people might think that there is a deeper connection between the theme and the mechanics of the particular game than I do. For example, they might think there is some genuine non-arbitrary connection between the mechanics of monopoly and the real estate market.
See the existing debate about ludonarrative dissonance.

If people overfocus on the theme, then they will make incorrect predictions about AI.

For example, they'll hear that AI has solved Full-Press Diplomacy and extrapolate that AI will soon be able to solve other games of a similar theme (i.e. international military negotiations).
However, they should instead extrapolate that AI will soon be able to solve other games with similar mechanics.
Here's some practical advice: imagine the game had the same mechanics but a different theme. Sure, AI has solved Full-Press Diplomacy, which is scary because the theme is militaries negotiating about nations to invade. But what if the theme was about gardeners negotiating which flowers to buy? Okay, still a bit scary, but you get my point.

Appendix

So anyway, what are game mechanics?

Is time discrete (i.e. turn-bas

...
The Geometric Expectation
by Scott Garrabrant — 2022-11-23T18:05:12.206Z

A Suspicious Pattern

There is a pattern that shows up in many of the toys we like to play with around here: the pattern of maximizing the expected logarithm.

Nash bargaining is a method for aggregating preferences without a means to directly compare them. When Nash bargaining, you are maximizing the expected logarithm of utility, where the expectation is over uncertainty about which person you are.

Kelly betting is an extremely useful tool for not putting all your future wealth in one basket. When Kelly betting, you are maximizing the expected logarithm of your wealth.

The log scoring rule is a very natural way to extract beliefs. When maximizing your log score, you are maximizing the expectation of the logarithm of the probability you assign to the right answer. This is one example of a general pattern. Maximizations of expected logarithms show up all over information theory, often phrased as minimizing the negative of the expected logarithm.

Why does maximization of the expected logarithm keep showing up?

One answer is that all of the instances of it showing up are actually related. In my previous two posts, I made some connections between Nash bargaining and Kelly betting. The fact that Kelly betting can be used to model Bayesian updating illustrates its relationship with the information theory applications. To a certain extent, there is really only one instance of this pattern.

However, I think that there is another argument for why you should expect this pattern to show up a lot, which is that the pattern is very simple. More simple than it looks on the surface. It only looks complicated because mathematicians have failed us.

The Geometric Integral

One of the most underrated concepts in mathematics is the geometric integral, given by ∏f(x)dx=e∫ln(f(x))dx. (The fact that I couldn't easily get a latex symbol that looks like an elongated P is a testament to its underratedness.) The geometric integral is just like the standard integral, but everywhere you would add, you multiply instead. Defining it in terms of the standard (arithmetic) integral with logs and exponents is insulting to its nature, and I don't recommend thinking of it that way. (You wouldn't define x×y as eln(x)+ln(y).) Instead, you should just think of it as the multiplicative version of the integral. However, using logs and exponentiation, it is the fastest way to get the definit...
"Far Coordination"
by DragonGod — 2022-11-23T17:14:41.830Z

Epistemic Status

Early and rough thoughts. Sharing because I believe the core idea is sound[1].

Introduction

What is "Far Coordination"?

Coordination across such vast distances of spacetime that communication between coordinating parties is physically impossible[2].

Why is Far Coordination Possible?

Premises

Assumptions on which the rest of this post lies:

Computationalism: all agents of interest are computations that can be replicated to arbitrarily high fidelity on other substrates
Robust cryptography
Parties can send messages to each other that can't be forged
Digital signatures can't be forged

Spacetime Invariance of Computation

Computation is time and space invariant[3].

"2+2" evaluates to 4 whether it's computed today, 5 billion years ago or 5 billion years from now. It also evaluates to 4 whether it's computed on earth, elsewhere in the Milky Way or somewhere in Andromeda.

The spacetime invariance of computation can be exploited to facilitate coordination across vast distances of time and space.

Why Care About Far Coordination?

I am interested in far coordination because I want our civilisation to become interstellar (and eventually intergalactic). Coordination across vast distances of spacetime would be necessary to remain a united civilisation (as opposed to fracturing into a multitude of [potentially very divergent] civilisations) after we venture out of the solar system.

I would prefer for earth originating civilisation to remain broadly united, should we venture to the stars.

Use Cases for Far Coordination

Preventing "Monstrous" Divergences

Some future-minded folks I've chatted with seem to be concerned that posthumans would diverge from us in considerable ways. Particularly, there's concern that their values may be so radically altered that they would seem to us (were we around to observe them) not just alien, but monstrous[4].

Policing Particular Behaviours

Some people are very concerned about astronomical suffering. For those with suffering-focused moral foundations, the prospect of astronomical suffering may dominate all considerations. There may be a desire to implement mechanisms to prevent anyone from running simulations of sentient beings for the purpose of torturing them. Such policing may be viable for a planetary or even interplanetary civilisation, but without means to coordinate behaviour across vast ...
Conjecture Second Hiring Round
by Connor Leahy — 2022-11-23T17:11:42.524Z

Conjecture is hiring! We have open roles for all teams, both technical and non-technical. We have written a bit more about the teams at Conjecture

here

and you can view all open positions

here

. Applications for the hiring round will close December 16.

Conjecture is an AI Safety startup that aims to solve the alignment problem.

At Conjecture, we focus on short timeline scenarios. We critically evaluate if the research we conduct actually cuts at alignment, and constantly seek to improve ourselves. We think the amount of work that needs to be done is immense, so we are working to build an organization capable of significantly scaling up and diversifying research directions.

We develop our own large models and research infrastructure. In order to maintain our ability to fund and conduct research at scale, we are also working on building AI driven products that we will soon be offering to a wider audience.

We are based in London, and are excited to be part of the growing alignment scene here. We’re in person by default, sponsor visas for all roles, and support employees’ relocation to London financially and logistically.

Nice views from the Conjecture office :)

For technical teams, the roles we’re most interested in filling are:

ML Engineering Lead
Security Lead
Research Engineer (Engineering Focus)
Research Engineer (Research Focus)
Product Engineer

We’re looking for candidates who can

hit the ground running

. Much of our research agenda at Conjecture involves working with the latest advancements in machine learning and working with large distributed computing clusters. While we expect driven people can skill up quickly in these areas, we will prioritize candidates who have backgrounds in high-performance computing, particularly those with a track record of owning projects end-to-end.

For non-technical teams, the roles we’re most interested in filling are:

Finance Lead
Strategy Analyst

Candidates don’t need to have prior experience with AI safety. More important is drive, professionalism, and a history of making real things happen in the world. While we have a vision for what these roles will look like, we are open-minded and are eager to hear from driven candidates who have concrete suggestions for actions/plans they’d like to execute alongside Conjecture.

For all roles, we’re looking for self-driven and flexi...
Conjecture: a retrospective after 8 months of work
by Connor Leahy — 2022-11-23T17:10:23.510Z

This post is a brief retrospective on the last 8 months at Conjecture that summarizes what we have done, our assessment of how useful this has been, and the updates we are making.

Intro

Conjecture

formed

in March 2022 with 3 founders and 5 early employees. We spent our first months growing the team, building infrastructure, exploring different

research agendas

,

running Refine, publishing our internal

infohazard policy

,

establishing an operational foundation for the business, and raising investments.

It’s been intense! For many of us at Conjecture, the last eight months have been the hardest we’ve worked in our lives. Working on such an immensely difficult problem as alignment alongside a team of brilliant and driven colleagues is, to say the least, galvanizing.

In some ways, this makes it difficult to step back and critically reflect on our work. It is easy to mistakenly measure progress by effort, and the last thing you want to hear after maxing out effort is that it wasn’t good enough.

However, reality does not grade on a curve. We need to advance significantly faster than traditional science in order to solve alignment on short timelines.

By this standard, the sober reflection is that most of our efforts to date have not made meaningful progress on the alignment problem. Our research has not revealed new methods that make neural networks more interpretable or resolve inner or outer alignment problems, and our coordination efforts have not slowed the pace at which AI capabilities are advancing compared to safety. When measured against p(Doom), our efforts haven’t

cut it

.

That’s not to say this work has been useless. We have learned a lot about where we went wrong, and made a number of changes that put us in a better position to make progress than we were in March. Measuring ourselves against a high standard enables us to constantly improve and be realistic about the difficulty of the problem ahead of us.

The reason we are writing this reflection is to calibrate ourselves. We do not want to be seen as cutting alignment if we are not. What matters is that we ground ourselves in reality and make public as many of our efforts (and mistakes!) as possible in order to gather feedback and update quickly.

What we have done and how useful we think it is

Infrastructure

We have built our own infrastructure to deploy large language models and do be...