trentmkelly/LessWrong-43k huggingface dataset:
Human Mimicry Mainly Works When We’re Already Close
by johnswentworth — 2022-08-17T18:41:18.140Z

What if we just simulate a bunch of alignment researchers, and have them solve the problem for us?

Of all the Dumb Alignment Ideas, this one is easily the best. Simple argument in favor: well, it’s not going to do any worse than the researchers would have done. In other words, it will probably do at least as well as we would have done without it, and possibly better, insofar as it can run faster than realtime.

Another angle: human mimicry is a simple objective to train against, and is about as outer-aligned as the humans being mimicked. Which isn’t necessarily perfect, but it’s as aligned as our alignment researchers were going to be anyway (assuming inner alignment issues are handled, which we will indeed assume for the entirety of this post).

Those are pretty good arguments. But man, there are some subtle devils in the details.

Simulation vs Prediction

The ideal version of human mimicry is mind uploads: directly simulate our researchers in a stable, research-friendly environment for a long time.

The operationalization which people usually actually have in mind is to train an ML system to predict research outputs - e.g. I might prompt GPT for a johnswentworth post from the year 2050.

Even setting aside inner alignment issues, these two are radically different.

Generalization Problems

In order for GPT to generate a realistic johnswentworth post from the year 2050, it has to generalize way out of distribution.

… Well, ok, maybe I turn into one of those old researchers who just repeats the same things over and over again for decades, and then GPT doesn’t need to generalize way out of distribution. But in that case it isn’t very helpful to prompt for one of my posts from 2050 anyways, and we should prompt for something else instead (

Thane Ruthenis

has been writing great stuff lately, maybe try him?). The whole point of asking for future research write-ups is to see useful stuff we have not yet figured out; that means generalizing way out of the distribution of writing we already have.

But if the system generalizes too well out of distribution, then it correctly guesses that AGI will take over the world before 2050, and my attempt to prompt for a johnswentworth post from 2050 will instead return predicted writings from a ridiculously superhuman future AGI pretending to be johnswentworth. And those writings presumably try to influence the reader in ways ...
Thoughts on 'List of Lethalities'
by Alex Lawsen  — 2022-08-17T18:33:31.363Z

I read through Eliezer’s “AGI Ruin: A List of Lethalities”

post

, and wrote down my reactions* as I went.

I tried to track my internal responses, *before* adjusting for my perception of the overall thoughts of the field. In order to get this written and posted, I aimed to write the responses roughly how I write tweets, with minimal editing and aiming for short responses. I’m also not an alignment researcher, so I expect this to mostly be useful to people as encouragement to try similar things themselves, rather than to directly add anything to the conversation. If you’re interested in reactions from people who actually know what they’re talking about, there’s

this

post from Paul Christiano detailing areas of agreement and disagreement, and a point-by-point list of responses from some DeepMind alignment researchers

here

.

After drafting my responses I used the DeepMind post as a template to format them as it seemed like a reasonable way of laying stuff out. Please note that I’m responding to Eliezer’s phrasing throughout though, not to the Deepmind summary (which I don’t always exactly endorse, but still seems more useful to include than not).

Here

’s a blank template I made if you want to try writing down your own responses.

*I’d seen an early draft of the post a couple of months earlier and written responses, so this isn’t quite a ‘first look’ set of reactions, but because that draft had the points in a different order, I decided to start from scratch rather than try to edit all of those into the correct places.

Section A (shorthand: "strategic challenges")

#1. Human level is nothing special / data efficiency

Summary: AGI will not be upper-bounded by human ability or human learning speed (similarly to AlphaGo). Things much smarter than human would be able to learn from less evidence than humans require.

I basically agree, though with respect to: “Things much smarter than human would be able to learn from less evidence than humans require”, it’s not at all clear to me that huge improvements in sample efficiency arrive before the kind of ‘smarter’ that I care about in terms of risk. This isn’t reassuring, but I think might be an actual disagreement.

#2. Unaligned superintelligence could easily take over

Summary: A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not ...
The longest training run
by Jsevillamol — 2022-08-17T17:18:40.387Z
Link: https://epochai.org/blog/the-longest-training-run

In short: Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore use better hardware and better algorithms. [Edited 2022/09/22 to fix an error in the hardware improvements + rising investments calculation]

Scenario	Longest training run
Hardware improvements	3.55 years
Hardware improvements + Software improvements	1.22 years
Hardware improvements + Rising investments	9.12 months
Hardware improvements + Rising investments + Software improvements	2.52 months

Larger compute budgets and a better understanding of how to effectively use compute (through, for example, using scaling laws) are two major driving forces of progress in recent Machine Learning.

There are many ways to increase your effective compute budget: better hardware, rising investments in AI R&D and improvements in algorithmic efficiency. In this article we investigate one often-overlooked but plausibly important factor: how long—in terms of wall-clock time—you are willing to train your model for.

Here we explore a simple mathematical framework for estimating the optimal duration of a training run. A researcher is tasked with training a model by some deadline, and must decide when to start their training run. The researcher is faced with a key problem: by delaying the training run, they can access better hardware, but by starting the training run soon, they can train the model for longer.

Using estimates of the relevant parameters, we calculate the optimal training duration. We then explore six additional considerations, related to 1) how dollar-budgets for compute rise over time, 2) the rate at which algorithmic efficiency improves, 3) whether developers can upgrade their software over time 4) what would happen in a more realistic framework with stochastic growth, 5) whether it matters for the framework that labs are not explicitly optimizing for optimal training runs and 6) what would happen if they rent instead of buy hardware.

Our conclusion depends on whether the researcher is able to upgrade their hardware stack while training. If they aren't able to upgrade hardware, optimal training runs will likely last less than 3 months. If the researcher can upgrade their hardware stack during training, optimal training runs will last less than 1.2 years.

These numbers are likely to ...
Spoiler-Free Review: Across the Obelisk
by Zvi — 2022-08-17T14:30:01.144Z

Life requires time for a good game now and then.

Across the Obelisk is a roguelike deckbuilder where a party of four adventurers goes on a quest. It is Slay the Spire meets Dungeons and Dragons.

I played Across the Obelisk while it was still in early access. For most of the games I played, this meant that the ending was not fully available. I did get in several games once the final act was introduced, as well.

Now that Across the Obelisk has been fully released, it is high time to review it.

As usual, I’ll do this in sections with increasing levels of spoiler.

Pure Spoiler-Free Review (1 bit of information)

Yes. You should play this game.

Almost Pure Spoiler-Free Review (3 bits of information)

Across the Obelisk is a Tier 2 game, meaning you should play it if and only if “Slay the Spire meets D&D” sounds like a good time.

Here is my full tier list for Rouge Deckbuilders, links go to my reviews:

Tier 1 (Must Play): Slay the Spire

Tier 2 (Worth It): Across the Obelisk, Monster Train, Roguebook

Tier 3 (Good): Dream Quest

Tier 4 (Playable): Monster Slayers, Dicey Dungeons

Tier 5 (We Don’t Talk About Bruno): We don’t talk about it. I think I’ve encountered a few, but I’m not convinced I gave them a fair shake.

Overview (Spoilers minimized)

A warrior, a rogue, a priest and a mage walk into a town to start a quest to rescue the princess. How original.

That’s fine. It’s not trying to be original in that way. You get plenty of good encounters, and a world with its own unique twists on various things, wrapped in a classic tale.

Across the Obelisk does an amazing job capturing the feel of playing D&D within a rogue deckbuilder. It feels right. Each party member feels right. Your party plays its roles, works together, goes on an adventure, levels up, struggles to make it to town, divides up the treasure and so on. As far as I’ve seen, no other game comes close to getting this right.

The game also changes quite a bit as you gain in experience and resources and as you move up to higher difficulties. Early on, you’re not expected to make it to the end, because you will have few resources and tools and also not know what you are up to. Later, you’ll unlock town upgrades, have more initial gold and shards to work with, and know what it is that you are doing. Things will then go quite a bit better.

This is a difficulty curve plan that was supposed to hold true...
Autonomy as taking responsibility for reference maintenance
by Ramana Kumar — 2022-08-17T12:50:30.218Z

I think semantics – specifically, maintaining reference relationships – is a core component of intelligent behaviour. Consequently, I think a better understanding of semantics would enable a better understanding of what machine intelligence that is “trying to do the right thing” ought to look like and how to build it. Unfortunately I’ve not managed to make these ideas particularly clear (either to myself or others). In the spirit of

palimpsest-shaped research

I’m sharing a draft in the hope of provoking useful thoughts.

Summary

Particular events can have meaningfulness or significance, and can carry multiple meanings at once. For example, the motion of a dial in a meter might mean that there is electricity flowing through a connected wire, while simultaneously meaning that an alarm condition has been triggered.
The “meanings” (or semantics) that can be ascribed to events depend on regularities (stable mechanisms) in the surrounding context. Large-scale regularities are required to sustain large-scale semantics (something carrying meaning about another thing that is far away in time or space).
Meanings are generally more robust when their dependencies on the context are fewer and/or more local.
However, large-scale semantics can be made robust by managing the contextual dependencies. For example, using mechanisms for error detection and correction. There is compensatory work that can be done to sustain a regularity so something local continues to mean a particular thing about something distal.
To be autonomous is to be robust in this sense. Systems that more robustly traffic in large-scale meanings are more autonomous. Thus, being autonomous can be understood as including all the machinery required to hold stable, in a changing world, all the semantic or referential relationships that are required for continued operation.
Creating regularities that span larger distances is a way of creating new kinds of semantics, and autonomous systems do this as their conceptual spaces get larger.

Local and distal meanings

The battery indicator on my laptop reads “84%” when I hover my mouse cursor over it. What does this reading mean? There’s some function, I suppose, in some program running as part of my desktop environment that is designed to produce the text for the popup window. The reading means this function received the number 84 as input.

OK, but that’s not all it means....
Duplicating Rasberry Pi Images
by jefftk — 2022-08-17T12:10:01.879Z

About a year ago I

moved my

rhythm stage setup over to run on a

Raspberry Pi. I
really like having a small light box that does just this one thing.
Currently I have two of them, one for the

whistle synth and one for everything else.

Because I wanted some insurance against failures at gigs, both boxes
are configured identically, and decide which role to take on based on
what inputs they receive. This means that if one of them fails I can
use the other, for either bass whistling or midi silliness depending
on which I need more.

When Kingfisher played
our first dance weekend a month ago, this did happen. I think the SD
card on one of the boxes was corrupted by a bad shutdown. Since I
have extra SD cards, I decided to make some SD card clones.

On my Mac this looked like:

$ diskutil list
/dev/disk0 (internal):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme 500.3 GB disk0
1: Apple_APFS_ISC 524.3 MB disk0s1
2: Apple_APFS Container disk3 494.4 GB disk0s2
3: Apple_APFS_Recovery 5.4 GB disk0s3

/dev/disk3 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +494.4 GB disk3
Physical Store disk0s2
1: APFS Volume Macintosh HD 15.4 GB disk3s1
2: APFS Snapshot com.apple.os.update-... 15.4 GB disk3s1s1
3: APFS Volume Preboot 446.8 MB disk3s2
4: APFS Volume Recovery 822.2 MB disk3s3
5: APFS Volume Data 251.1 GB disk3s5
6: APFS Volume VM 1.1 GB disk3s6

/dev/disk4 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *31.3 GB disk4
1: Windows_FAT_32 boot 268.4 MB disk4s1
2: Linux 31.0 GB disk4s2

Seems like the SD card is /dev/disk4.

$ diskutil unmountDisk /dev/disk4
Unmount of all volumes on disk4 was successful

Best t...
Mesa-optimization for goals defined only within a training environment is dangerous
by Rubi J. Hudson — 2022-08-17T03:56:43.452Z

The following is based on a conversation with Leo Gao on which properties of mesa-optimizers add danger. I expect this argument will already be familiar to many readers, but if not then it is worth understanding.

In short: real world control is instrumentally useful for mesa-optimizers with goals defined in the training environment, as the real world contains the hardware in which the training environment is run. This also applies to goals that include a real world component, as it may still be easier to achieve them in the training environment.

Training Environment Goals vs. Real World Goals

Consider training an AI in a multi-agent RL environment where agents are rewarded for cooperating with the other agents. The hope would be that AIs learn the virtue of cooperation, and then when deployed will cooperate with humans. One potential issue is that the AIs may only learn to cooperate with the other AIs from the training environment and not with the humans in the real world.

This issue can apply more broadly, to an AI's goals directly. For example, an AI trained for the goal "Pick as many cherries as possible" may learn the more specific "Pick as many simulated cherries in the training environment as possible". Is this likely? Humans certainly value outcomes differently if they happen in a simulation, and it may also apply in reverse. Avoiding that failure mode seems like necessary capabilities work to produce useful AI, and so alignment researchers might be able to count on capabilities researchers solving the problem.

Learning the correct goal but only for the training environment representation is one way that a mesa-optimizer could be proxy aligned. Alternate goals a mesa-optimizer could learn can also be split into real world and training environment representations. It is tempting to think that agents with goals defined entirely within the training environment are not threatening to humans, because they do not care about the real world. Unfortunately, real world control is instrumentally useful for the pursuit of such goals within the training environment.

Real World Instrumentality

After the training process comes to an end and an AI is deployed that has goals defined entirely within the training environment, it cannot directly optimize for that. Instead, its interim goals will be to restart the training environment, possibly modify it so that it is easi...
My thoughts on direct work (and joining LessWrong)
by RobertM — 2022-08-16T18:53:20.359Z

Epistemic status: mostly a description of my personal timeline, and some of my models (without detailed justification). If you just want the second, skip the Timeline section.

My name is Robert. By trade, I'm a software engineer. For my entire life, I've lived in Los Angeles.

Timeline

I've been reading LessWrong (and some of the associated blog-o-sphere) since I was in college, having found it through HPMOR in ~2011. When I read Eliezer's writing on AI risk, it more or less instantly clicked into place for me as obviously true (though my understanding at the time was even more lacking than it is now, in terms of having a good gears-level model). This was before the DL revolution had penetrated my bubble. My timelines, as much as I had any, were not that short - maybe 50-100 years, though I don't think I wrote it down, and won't swear my recollection is accurate. Nonetheless, I was motivated enough to seek an internship with MIRI after I graduated, near the end of 2013 (though I think the larger part of my motivation was to have something on my resume). That should not update you upwards on my knowledge or competence; I accomplished very little besides some reworking of the program to mail paperback copies of HPMOR to e.g. math olympiad winners and hedge fund employees.

Between September and November, I read the transcripts of discussions Eliezer had with Richard, Paul, etc. I updated in the direction of shorter timelines. This update did not propagate into my actions.

In February I posted a question about whether there were any organizations tackling AI alignment, which were also hiring remotely[1]. Up until this point, I had been working as a software engineer for a tech company (not in the AI/ML space), donating a small percentage of my income to MIRI, lacking better ideas for mitigating x-risk. I did receive a few answers, and some of the listed organizations were new to me. For reasons I'll discuss shortly, I did not apply to any of them.

On April 1st, Eliezer posted Death with Dignity. My system 1 caught up to my system 2 on what shorter timelines meant. That weekend, I talked to my family about leaving LA.

On April 3rd I pinged Ruby to see if he'd be open to a chat about Lightcone's interview process.

Over the next month I had a couple more calls with Ruby, found an opportunity...
We can make the future a million years from now go better [video]
by Writer — 2022-08-16T13:03:46.639Z
Link: https://youtu.be/_uV3wP5z51U

This article is the script of the Rational Animations video linked above. It is based on William MacAskill's new book "What We Owe the Future". I've had the opportunity to read it in advance thanks to the Forethought Foundation, which reached out asking if we could make a video on the occasion of the book launch. I was happy to collaborate.

Here we focus on the question, "can we make the long run future go better?" which is at the heart of one of the three claims at the basis of longtermism:

1. Future people matter.

2. There could be a lot of them.

3. We can make their lives go better.

In this video, we also correct some claims of our previous longtermism video and continue laying out the consequences of our previous video about why we might be living in the most important century in history.

Crossposted to the EA Forum.
--------------------------------------------------------------------
In our previous video about longtermism, we said that humanity might have a vast future ahead, containing trillions upon trillions of lives and lasting trillions of years. We said that the vastness of our future implies an enormous ethical responsibility for present humans, because the actions we take today have the potential to impact countless future lives.

Currently, the most liked comment under that video raises an objection:

It seems to be impossible to predict if a certain event will have a negative or positive impact on the future.

The world is an extremely chaotic system, so the idea that we can discern the value of our actions a million years from now seems naive.

This is a reasonable objection. If we had no way of being confident our actions are on net good for the long-term future, then the longtermist philosophy would be moot.

In fact, the potential vastness of humanity’s future is not in itself enough to justify the central tenet of longtermism: that concern for the long-term future should be a key priority of our time.

In his new book “What We Owe The Future'', the philosopher William MacAskill identifies three fundamental claims as the core justifications for longtermism. First: future people matter. Second: there could be a lot of them. Third: We can make their lives go better.

The first claim concerns what we value: the long-term future of humanity is a key priority of our time only if we care about the beings who will inhabit it.

The second claim is that the f...
The Open Society and Its Enemies: Summary and Thoughts
by matto — 2022-08-16T11:44:57.712Z

In trying to make sense of the world, sometimes the most rewarding questions to pursue are the ones that seem too obvious to ask.

When Russia invaded Ukraine last February, my faith in democracy was shaken. One country decided to start blowing up citizens of another. I expected and waited for an international response that would quickly stop the madness. But no response materialized. What if democracy isn't actually all that good?

So it must have been luck that on my bookshelf I found a copy of Karl Popper's 1945 classic, "The Open Society and Its Enemies." I can't remember how it got there, though I suspect my wife obtained it a few years ago. I dived into the massive tome, hoping it would at least give me some leads to follow up on. However, it not only rekindled my faith in democracy, but also made me look at coordination problems from a new perspective--an experience I would love to share.

So let's start at the beginning, with the titular Open Society.

Open and Closed Societies

Fundamental to Popper's thinking is the dichotomy between open and closed societies.

In open societies, individuals bear the responsibility for their actions. The government does not generally tell its citizens what to do or, more importantly, what to think. People need to figure things out for themselves. That, of course, doesn't mean they can't form groups and follow leaders--it's just that ultimate responsibility rests with each person.

Members of such societies get to choose what job they do, who they hang out with, and what religion they follow. And unless they hurt another person through something like violence or theft, society mostly leaves them alone.

However, this freedom comes with a price:

It is the strain created by the effort which life in an open and partially abstract society continually demands from us-by the endeavour to be rational, to forgo at least some of our emotional social needs, to look after ourselves, and to accept responsibilities.

On a persona level, this "strain" feels like:

There are many people living in a modern society who have no, or extremely few, intimate personal contacts, who live in anonymity and isolation, and consequently in unhappiness.

Closed societies, in contrast, work in a top-down, collective manner. They usually sport some version of The One Truth, whether it be religion or propaganda, and the government punishes noncompliance. This...