trentmkelly/LessWrong-43k huggingface dataset:
Solving the whole AGI control problem, version 0.0001
by Steven Byrnes — 2021-04-08T15:14:07.685Z

The bridge to AGI control. Not quiiiiite ready for rush-hour traffic… Mind the gaps!!

(image source)

(Update: Note that most of the things I wrote in this post are superseded (or at least explained better) in my later “Intro to Brain-Like-AGI Safety” post series.)

I spend most of my time on relatively straightforward tasks—”straightforward” in the sense that I know how to proceed, and have some confidence that I can make progress, and making progress will probably be helpful. This is all very satisfying. But then I’m also trying to spend (at least) one day a week trying to “solve the whole AGI control problem”. (Specifically, the problem described in:

My AGI Threat Model: Misaligned Model-Based RL Agent

, which to be clear is still only one part of Safe & Beneficial AGI.)

So yeah, one day a week (at least), it’s big-picture end-to-end thinking, no excuses. My first couple attempts involved a lot of staring at a blank screen, muttering to myself "oh god oh god oh god, every idea is terrible, I'm in way over my head, we’re doomed…". But then I figured, it would be more useful to write up my current thoughts on the whole landscape of possible approaches, as an opportunity to clarify my thinking and get other people’s pushback. I'm hoping to repeat this exercise periodically.

(I’d love it if more people did this too FWIW.)

If I'm unjustifiably dismissing or omitting your favorite idea, try to talk me into it! Leave a comment or let’s chat.

Relatedly, if anyone feels an urge to trust my judgment (for reasons I can't fathom), this is an especially bad time to do so! I very much don’t want this brain-dump of a post to push people towards or away from what they think is a good direction. I haven’t put much thought into most parts of this. And what the heck do I know anyway?

Intended audience: People familiar with AGI safety literature. Lots of jargon and presumed background knowledge.

No claims to originality; many thanks to everyone whose ideas I’m stealing and butchering.

1. Corrigibility

(In the broad

Paul sense

of “the system is trying to do what the supervisor wants it to try to do”.)

1.0 Is corrigibility a worthwhile goal?

I think this is more-or-less uncontroversial, at least in the narrow sense of “knowing how to make a corrigible AGI would be a helpful thing to know how to do”. Well, Eliezer has argued (e.g.

this comment

) that corrigible motivation might not b...
How should I behave ≥14 days after my first mRNA vaccine dose but before my second dose?
by dsj — 2021-04-08T06:24:28.213Z

I am due to be vaccinated against Covid today. Let us suppose that I receive either the Pfizer/BioNTech (BNT162b2) or Moderna (mRNA-1273) vaccine, and subsequently receive my second dose on schedule (21 days later for BNT162b2; 28 days later for mRNA-1273).

Data from phase III trials for these vaccines suggest substantial efficacy at preventing disease starting at 14 days after the first dose:

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine, Correspondence from Danuta M. Skowronski, M.D.

Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine, Figure 3

While confidence intervals are wider due to the relatively small number of observation days, the BNT162b2 vaccine appears to be 92.6% effective in the corresponding 14- to 20-day window, while the mRNA-1273 vaccine appears to be 94.2% effective in the corresponding 14- to 27-day window. This is comparable to their efficacies even ≥14 days after the second dose.

Data for severe disease is also promising, though much sparser and hence more difficult to interpret.

On the other hand, at least one large observational study estimates only 57% efficacy against symptomatic Covid-19 for BNT162b2 in the 14- to 20-day window, as compared to 94% ≥7 days after the second dose. Other measures (documented infection, hospitalization, and severe disease) are similarly worse during that window.

In my case, I’m planning on air travel within the continental United States relatively soon, and while I could postpone it several additional weeks if necessary to protect myself and other travelers from severe disease, I prefer not to if there’s no advantage. So, which studies provide the most important evidence about the relative risk of “returning to society” 14 days after my first dose, as opposed to waiting until 7 or even 14 days after my second dose?
Is there any plausible mechanisms for why taking an mRNA vaccine might be undesirable for a young healthy adult?
by Mati_Roy — 2021-04-08T01:18:42.434Z

We haven't seen any problem in the short term, but is the probability of long term negative side effects (given it's a new technology) sufficiently unlikely for a young healthy person to take it?

Am particularly concerned with unknowns long-term future negative side effects, but which we might be able to estimate from first principles, or through some more abstract reference class, or something.

I already fully isolate, and would only value my freedom a bit, whereas I value my health a lot.
Notes on Perseverance
by David Gross — 2021-04-08T00:01:36.105Z

This post examines the virtues of perseverance, persistence, resilience, grit, fortitude, tenacity, sisu, and others in that bailiwick. It is meant mostly as an exploration of what other people have learned about these virtues, rather than as me expressing my own opinions about them, though I’ve been selective about what I found interesting or credible, according to my own inclinations. I wrote this not as an expert on the topic, but as someone who wants to learn more and to become better at it. I hope it will be helpful to people who want to know more about these virtues and how to nurture them.

What are these virtues?

People who exhibit these virtues rebound from setbacks and are not easily discouraged. They are more apt to overcome challenges than to be overcome by them.

Some of these virtue-words emphasize coping-with-adversity (resilience, fortitude), others emphasize continued effort (tenacity, perseverance, persistence), and others are a combination of those and other things (grit, sisu).

“Grit” in particular has been investigated under the positive psychology banner in recent years, and so deserves some extra attention here. As the researchers define it, grit is a combination of perseverance and passion. Perseverance in turn is defined as consistency of effort, while passion, less intuitively, is defined as consistency of interests. (Passion, in other words, has less to do with the level of present enthusiasm than with sustained interest over the long term. Researchers measure passion in people by asking them how little they identify with statements like “new ideas and projects sometimes distract me from previous ones,” or “my interests change from year to year.”[1]) Grit is correlated with (and, some critics say, difficult to distinguish from) “conscientiousness” in the “big five” model of personality.

“Sisu” seems to be a sort of in extremis perseverance: persistence to the bitter end in spite of terrible odds, when you feel you have already given your all.[2]

Related virtues

Sometimes these virtues are accompanied by a mental attitude of determination, resolve, steadfastness, firmness, or duty.

A related virtue that is more passive but has a similar sense of putting up gracefully with adversity is endurance. Another along the same lines is patience/forbearance.

Fortitude and resilience can be assisted by toughness, hardiness, and strength.

The virtues of re...
Open and Welcome Thread - April 2021
by Raemon — 2021-04-07T21:15:11.800Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Another (outer) alignment failure story
by paulfchristiano — 2021-04-07T20:12:32.043Z

Meta

This is a story where the alignment problem is somewhat harder than I expect, society handles AI more competently than I expect, and the outcome is worse than I expect. It also involves inner alignment turning out to be a surprisingly small problem. Maybe the story is 10-20th percentile on each of those axes. At the end I’m going to go through some salient ways you could vary the story.

This isn’t intended to be a particularly great story (and it’s pretty informal). I’m still trying to think through what I expect to happen if alignment turns out to be hard, and this more like the most recent entry in a long journey of gradually-improving stories.

I wrote this up a few months ago and was reminded to post it by Critch’s

recent post

(which is similar in many ways). This story has definitely been shaped by a broader community of people gradually refining failure stories rather than being written in a vacuum.

I’d like to continue spending time poking at aspects of this story that don’t make sense, digging into parts that seem worth digging into, and eventually developing clearer and more plausible stories. I still think it’s very plausible that my views about alignment will change in the course of thinking concretely about stories, and even if my basic views about alignment stay the same it’s pretty likely that the story will change.

Story

ML starts running factories, warehouses, shipping, and construction. ML assistants help write code and integrate ML into new domains. ML designers help build factories and the robots that go in them. ML finance systems invest in companies on the basis of complicated forecasts and (ML-generated) audits. Tons of new factories, warehouses, power plants, trucks and roads are being built. Things are happening quickly, investors have super strong FOMO, no one really knows whether it’s a bubble but they can tell that e.g. huge solar farms are getting built and something is happening that they want a piece of. Defense contractors are using ML systems to design new drones, and ML is helping the DoD decide what to buy and how to deploy it. The expectation is that automated systems will manage drones during high-speed ML-on-ML conflicts because humans won’t be able to understand what’s going on. ML systems are designing new ML systems, testing variations, commissioning giant clusters. The financing is coming from automated systems, the clu...
Which counterfactuals should an AI follow?
by Stuart_Armstrong — 2021-04-07T16:47:42.505Z

Thanks to Rebecca Gorman for her help with this post.

In the past, I've talked about counterfactual incentives. In that, the agent was motivated to maximise not an actual reward, but the (counterfactual) reward it would have gotten if things had been different.

"If things had been different" is ambiguous, though - different in what way, and with what consequences? In that post, I assumed an "evidential" counterfactual, but there are other ways of defining a counterfactual, and they have different consequences. This is important for designing agents that can use counterfactual reasoning for AI safety.

So this post will look at four kinds of counterfactuals, termed "causal" (Pearl's "do" operator), "evidential" (standard conditioning), "closest world", and "closest model", and see how they differ.

How do humans understand counterfactuals?

Imagine you are a jury-member in the trial of one RM, accused of killing one Tybalt C. Your role is to weigh up the evidence, including your impressions of the character of RM and any of the witnesses.

At one point in the trial, a journalist giving evidence blurts out:

"But RM was involved in a street brawl with the Capulet gang!"

The judge immediately interrupts, and says:

"That conviction is spent. The jury will disregard the implications of that statement, and act as if RM was not involved in the street brawl."

If we wanted to obey the judge's instructions, how should we do it? They obviously want us to reason as if we were in a counterfactual universe, but which counterfactual?

Initial model

Let's model the situation in the following way. You have to decide whether RM is innocent. You've seen some evidence at the trial. You've heard the journalist's statement and the judge's response. You wonder whether RM is of bad character (which is very relevant to assessing his guilt). You also wonder whether RM was involved in the brawl with the Capulets or not.

All this can be encoded in a causal graph, where each node is a true/false boolean:

The pale pink nodes are the ones that you've personally observed; the blue nodes are the ones that you're inferring. The arrows denote the causal structure.

The red arrow denotes that, if the source node is true, then the target node is false (with probability 99.9%). So if RM is of bad character, then, based on the other evidence you've seen, he's almost certainly guilty.

Is RM of bad character?...
If my previous research is wrong, what are my options ?
by BB6 — 2021-04-07T16:44:28.156Z

In 2007 I coauthored a scientific paper, which is essentially a method for food analysis. To be honest, we have noticed one sample behaving weirdly already at that time, but we rationalised it away and omitted the weird sample from the results. Still, just in case, we have burried the article in a local journal of our institution, which is low profile. Later I discovered, it is wrong by design (My design :-( ). Knowing more background would have prevented the problem, but I did it as a side job. I discovered the full stupidity of the design in cca 2012 and told my boss. He suggested we correct ourselves by publishing another paper, claiming we did more experiments and found out we were wrong. I agreed, obtained more samples, my colleague worked on it, but I went on a long maternal leave with 2 kids and forgot to follow up on it. It is uncorrected up until this day. What are my options ?

If we publish a new article, as my boss wanted, I fear some people will still find the first paper and not the second one, will keep quoting it, and, god forbid, use that published method. (To my best knowledge the method was not used so far. I hope it really wasn't). Retraction would be the cleanest way to do it, but also the worst looking. Also we could publish erratum. Erratums are paired up with the original articles and search engines show them together. But can erratum contain new samples ? Any other ideas ?

If my boss finds the correction a low priority, as he did so far, can I somehow act alone ?

Can I have any problems, that I did not correct myself fast enough ? I left some traces about knowing the design is wrong in 2012. I wrote the truth to one lady requesting the reprint. I also wrote a book chapter, explaining how important it is to know the backround while designing analysis method, and giving the example of this tricky food. I did not mentioned nor quoted my article...

Please, don't be too judgemental, and tell me, how you would or did handle these situations.
Alignment Newsletter Three Year Retrospective
by Rohin Shah — 2021-04-07T14:39:42.977Z

It has now been just shy of three years since the first Alignment Newsletter was published. I figure it’s time for an update to the

one-year retrospective

, and another very short

survey

. Please take the

survey

! The mandatory questions take just 2 minutes!

This retrospective is a lot less interesting than the last one, because not that much has changed. You can tell because I don’t have a summary or key takeaways, and instead I’m going to launch into nitty gritty details.

Newsletter stats

We now have 2443 subscribers, and tend to get around a 39% open rate and 4% click through rate on average (the click rate has higher variance though). In the one-year retrospective, I said 889 subscribers, just over 50% open rate, and 10-15% click through rate. This is all driven by organic growth; there hasn’t been any push for publicity.

I’m not too worried about the decreases in open rate and click rate:

I expect natural attrition over time as people’s interests change. Many of these people probably just stop opening emails, or filter them, or open and immediately close the emails.
In absolute terms, the number of opens has gone way up (~450 to ~950).
My summaries have gotten more pedagogic (see below), so people might feel less need to click through to the original.
I now summarize fewer items, so there are fewer chances to “catch people’s interests”.
We haven’t done any publicity, which I would guess is a common way to boost open rates (since newer subscribers are probably more likely to open emails?)

There was this weird thing where at the beginning of the pandemic, open rates would alternate between < 20% and > 40%, but would never be in between. I have no idea what was going on there.

I was also a bit confused why we’re only at #145 instead of #157, given that this is a weekly publication -- I knew I had skipped a couple of weeks but twelve seemed like too many. It turns out this newsletter was published every fortnight during the summer of 2019. I had no memory of this but it looks like I did take steps to fix it -- in the

call for contributors

, I said:

I’m not currently able to get a (normal length) newsletter out every week; you’d likely be causally responsible for getting back to weekly newsletters.

(This was probably true, since I did get back to weekly newsletters after getting new contributors!)

Changes

My overall sense is that the newsletter has been pretty stable and...
2020 predictions - Politics, Econ, Tech
by Bucky — 2021-04-07T14:10:40.338Z

The results are in! Here are my thoughts on the non-COVID set of my predictions. (For detailed thoughts on the COVID predictions see Zvi's analysis)

Questions, predictions, results and analysis below. Predictions are given as (Scott:Zvi:Me). Questions are strike-through if they didn't happen.

My analysis is about whether my predictions were reasonable given what I knew. It's hard to know whether Zvi & Scott could have done better as we didn't have the same information as each other. One obvious place this shows up is in UK/US questions. I'm in the UK so should have higher confidence on UK stuff and lower on US stuff.

Given the corrected result on q35 (see below), my score on this set of questions was marginally higher than Zvi who was marginally higher than Scott (S=1.53, Z=1.80, B=2.07) using Scott's scoring system. Unsurprisingly, I got a better log score on UK and worse on US that Zvi and Scott. The results are close enough that it doesn't really tell us that much but I'm at least happy that it wasn't a repeat of the COVID result!

(Note: I didn't predict on all of the questions)

POLITICS

21. Democrats nominate Biden, and he remains nominee on Election Day: (90:90:90)

Still seems reasonable

…

26. Trump is re-elected President: (50:50:60)

I outside-viewed the US election predictions.

I probably over-relied on my reference class (recent incumbents) here and could have adjusted better for Trump’s poor approval ratings. I adjusted slightly for this but not enough.

One commentor on Zvi's prediction post suggested that this should have been even lower than 50% which seems reasonable in hindsight.

27. Democrats keep the House: (70:70:60) &

28. Republicans keep the Senate: (50:60:60) (judged inconclusive)

Not something I knew much about so just assumed slightly more likely to keep than to switch. I presumably could have done better looking at the markets but that seems like cheating.

29. Trump approval rating higher than 43% on June 1: (30:40:50)

I was too high here, I think I was a bit lazy at looking at his ratings – I kinda went “oh he’s about 43% now, I guess he’ll be about the same. However he was on about 42% so I should have predicted under 50% really.

30. Biden polling higher than Trump on June 1: (70:80:70)

Still seems reasonable, Zvi was maybe justified in being higher but I don't think I could have put more than 70%

…

33. Boris still UK PM: (90:80:...
I Trained a Neural Network to Play Helltaker
by lsusr — 2021-04-07T08:24:27.982Z

In my experience, the most tedious parts of every machine learning project is annotating training data for my model. It would be nice if I could play a videogame with a keylogger and then train a neural network to copy me. That way my computer could play videogames for me without me having to manually annotate anything.

For my first experiment, I picked Helltaker, a game about a guy who goes to Hell to seduce demons. Helltaker is an easy game for a machine to play because:

It has a small, finite number of game states.
It uses static sprites instead of a 3D environment.
All the information necessary to play the game is present on the screen at all times; I could avoid adding any sort of recurrence into my neural network.

I wrote a data collection program to record keystrokes and video. I played each level four times. I fed the data into a convolutional neural network. I wrote another program that took screenshots of my screen, fed them into the neural network and translated the network outputs into keystrokes. You can see the model's keystrokes and confidences in the upper-left corner of the video below.

My neural network has—at one point or another—completed most (possibly all) of the puzzles and dialog trees. It utterly failed to survive the real-time section of Level IX.

In this particular video, my bot failed simply because my computer is not fast enough to run both the neural network and my OBS screen recording software at the same time. Deeper issues persist even when I turn OBS off.

The Waiting Problem

My neural network uses no recurrence. It is theoretically stateless. Given an image of the screen, it will always make the same decision. If my model looks at a turn-based puzzle and choose to wait then my model will continue to wait forever. To solve this problem, I told my model to never wait. No matter what it sees, the model always outputs a keystroke.

Never waiting is a good strategy for Helltaker's turn-based puzzles. Never waiting is a problem in the real-time section of Level IX. On Level IX there are situations where waiting is the right thing to do. This inability to ever wait contributed to my model's failure on Level IX. It frequently ran down to the spikes at the bottom and died.

A model should never wait in a turn-based game. A model should sometimes wait in a real-time game. I could fix this problem by manually dividing the turn-based sections ...