trentmkelly/LessWrong-43k huggingface dataset:
Seeking Power is Convergently Instrumental in a Broad Class of Environments
by TurnTrout — 2021-08-08T02:02:18.975Z

Edit, 5/16/23: I think this post is beautiful, correct in its narrow technical claims, and practically irrelevant to alignment. This post presents an unrealistic picture of the role of reward functions in reinforcement learning, conflating "utility" with "reward" in a type-incorrect fashion. Reward functions are not "goals", real-world policies are not "optimal", and the mechanistic function of reward is (usually) to provide policy gradients to update the policy network.

I expect this post to harm your alignment research intuitions unless you've already inoculated yourself by deeply internalizing and understanding Reward is not the optimization target. If you're going to read one alignment post I've written, read that one.

Follow-up work (Parametrically retargetable decision-makers tend to seek power) moved away from optimal policies and treated reward functions more realistically.

A year ago, I thought it would be really hard to generalize the power-seeking theorems from Markov decision processes (MDPs); the MDP case seemed hard enough. Without assuming the agent can see the full state, while letting utility functions do as they please – this seemed like asking for trouble.

Once I knew what to look for, it turned out to be easy – I hashed out the basics during half an hour of conversation with John Wentworth. The theorems were never about MDPs anyways; the theorems apply whenever the agent considers finite sets of lotteries over outcomes, assigns each outcome real-valued utility, and maximizes expected utility.

Thanks to Rohin Shah, Adam Shimi, and John Wentworth for feedback on drafts of this post.

Instrumental convergence can get really, really strong

At each time step t, the agent takes one of finitely many actions at∈A, and receives one of finitely many observations ot∈O drawn from the conditional probability distribution E(ot∣a1o1…at), where E is the environment.Footnote: environment There is a finite time horizon T. Each utility function u:OT→R maps each complete observation history to a real number (note that u can be represented as a vector in the finite-dimensional vector space R|O|T). From now on, u-OH stands for "utility function(s) over observation histories."

First, let's just consider a deterministic environment. Each time step, the agent observes a black...
Delta, Deaths, and Vaccinations
by jefftk — 2021-08-07T19:10:02.802Z

I've heard a lot about how the Delta variant is much more contagious
and how vaccination both reduces transmission and dramatically reduces
deaths, but I wanted to get a better sense of it by looking directly
at the data. As with

last
time, I pulled the data from

JHU's CSSE (

code). First, here's
the pandemic so far:

Cases (blue) and Deaths (orange), 7-day moving average, February
2020 to present. Deaths are
scaled 100x relative to cases, and then for each state both are scaled
to fit.

You can see how Northeastern states (NY, NJ, etc) had massively
disproportionate deaths in their first wave, though the statistics
can't tell us how much was undertesting vs overwhelming the medical
system vs older people being infected. You can also see that deaths
generally move along with cases, with a small lag.

Looking at the rightmost edge of each chart, you can see the effect
first of vaccination bringing cases down, and then Delta and bringing
them back up. Let's look at just the most recent 45 days, this
upswing, and sort the states by the vaccination status (CDC,
"Percent of Total Pop Fully Vaccinated by State of Residence"):

Cases per 1,000 population, 7-day moving average, last 45 days.

Since this is now in cases per 1,000 people, we can compare values
between states and not just curve shapes. Vaccination does seem to be
helping, since states with a larger portion of the population
vaccinated are seeing less increase. Still, it's roughly exponential
growth everywhere, including in high-vaccination states, as we can see
by scaling the charts:

Cases, 7-day moving average, last 45 days, scaled to fit.

Now, let's overlay deaths, again scaled 100x relative to cases:

Cases (blue) and Deaths (orange), 7-day moving average, last 45 days. Deaths are
scaled 100x relative to cases, and then for each state both are scaled
to fit.

Based on past waves, you would expect deaths to be rising along with
cases. While we do see this in some states (AR), in most states deaths
are rising much more slowly than cases, or even holding steady. This
is really good to see! The pattern is stronger in states with higher
vaccination rates, which makes sense, since we expect the vaccines to
do more to prevent death than infection because (a) more vulnerable
people are disproportionately vaccinated and (b) that's what we saw in
studies.

Over...
Rage Against The MOOChine
by Borasko — 2021-08-07T17:57:43.858Z

[This was a review of Andrew Ng's Machine Learning course for a math and computer science guide I was making that ended up spiraling into a rant against MOOC's (massive open online course), I left it un-posted for a while but I think it remains mostly true what the problems are with the current MOOC paradigm.]

Course: Machine Learning, Andrew Ng

I started out excited for this course and left disappointed with a changed outlook on online learning.

I’ll start off saying that I liked the theory section in this course. I think Andrew did a good job of explaining the theory of why he was doing the things he was doing. I liked the explanations of supervised and unsupervised learning. The later chapters on recommender systems and principal component analysis were also very interesting.

But the problem is that other than a brief overview of the skills there was not much other depth to the course. The course glazed over math topics like how to find the partial derivative of the cost function for gradient descent. As somebody without a rigorous math background I was fine with that because I don’t know how to do partial differential equations. Then we got to programming in Octave, during the lecture he said it was better this way because it’s faster and it doesn’t really pick sides of which data science programming language you should use. The programming exercises being in octave could be overlooked if the programming exercises were good, but sadly they aren't.

The programming assignments don’t teach you anything, at least not anything I could at this point say is useful. Usually they are some form of transcribing the loss function of the model you are learning from its mathematical representation to it’s coding representation. With everything usually given to you as a hint at the top of the problem in comments, or it’s vectorized form mentioned in the document you are given to work through the problem.

Which made me feel like I was mostly doing this for each coding assignment:

It was trivial, and it felt trivial. All of the actual hard part of making the model was usually made for you. The data being collected, cleaned, visualized, loaded, split and then sent to a function where you just type the loss function pretty much as described. The only time I really ran into problems with getting code to run is when I didn’t ...
Next PEAR Meet-Up is 8/14 Oregon Park
by SmokeAndMirrors — 2021-08-07T15:14:37.107Z

Hello Aspiring Rationalists of Portland,

So we had another interesting meet-up on 7/31 where we talked about Personality Tests, the status of current Genetic Technology and IVF, the idea of General Intelligence, and using Prediction Markets to judge long term policy successes. Picts are linked :)

Congratulations!

To Chase on landing a job at Genomic Prediction! Hopefully he’ll still have enough time to continue to intrigue us with his research and insights into the field.

House Keeping!

Feedback Survey Link
https://docs.google.com/forms/d/1tvWLjwxUyWbLBmZuukkxIxceVR3DVHkVsdW9KFE3ZGM/edit?usp=drivesdk
So we are doing a quarterly survey on “what type of events would you like to do more of?” There is also a section for open suggestions, and you can always email me if you would like to give more direct feedback. Also, all the participants will be entered into a raffle which will award $20 to a charity of their choice at the end of August.

We are now called PEAR (Portland Effective Altruists and Rationalists)!
For the time being, we are combining the ACX and EA Meet-up groups into PEAR (Portland Effective Altruists and Rationalists). Many of our interests overlap, and we would love to build a larger community of sustained rationalism in the Portland area.

Future
We will be setting up some holiday rationalist events in the near future, a calendar, along with a website, and some other cool things :)

Next Meet-Up:

Please feel free to come even if you feel awkward about it, even if you’re not ‘the typical Rationalist/EA/ACX reader’, or even if you’re worried people won’t like you, etc. I have had so much fun these past couple weeks just meeting up, talking with curious people about curious things, and we would love to have you.

When: Saturday 8/14 at 6:00pm

Where: Oregon Park (Our table will have a banner saying PEAR Meet-Up)

Virtual: There will also be a google meet available virtually at https://meet.google.com/ehm-xrnb-cmp

What:

Board Games: the aumann game, codenames, zendo, monikers, among us, and whatever else. Food: drinks, cookies, and pizza

Possible Discussion: What is something that has changed your mind or shaped your perspective recently?

What to bring: Feel free to bring snacks, drinks, games, and friends.

If people are still around afterwards we could always go to an outdoor bar :D

Google Calendar link! https://calendar.google.com/event?action=TEMPLATE&...
Research agenda update
by Steven Byrnes — 2021-08-06T19:24:53.726Z

“Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” –DL Moody (allegedly)

(Related:

Solving the whole AGI control problem, version 0.0001

. Also related: all my posts.)

I’m five months into

my new job

! This here is a forward-looking post where I write down what I’m planning to work in the near future and why. Please comment (or otherwise

reach out

) if you think I’m prioritizing wrong or going in a sub-optimal direction for Safe & Beneficial AGI. That’s the whole reason this post exists!

(UPDATE 5 MONTHS LATER: I should have been clearer: This post is primarily the "learning about neuroscience" aspects of my research agenda. The real goal is to design safe & beneficial AGI, and my learning about neuroscience is just one piece of that. As it turns out, in the few months since I wrote this, the great majority of my time did not really involve making progress on any of the things herein.)

I’ll work backwards: first the intermediate “problems to solve” that I think would help for the AGI control problem, then the immediate things I’m doing to work towards those goals.

An arrow X→Y means “I think doing/solving X will be helpful for doing/solving Y”.

1. Intermediate problems to solve

1.1 “The 1st-Person Problem”

(See

here

.)

It’s easy (or at least tractable) to find or generate tons of labeled data with safety-critical concepts and situations. For example, we can easily find a YouTube video where Alice is deceiving Bob, and label it "deception".

But is that useful? The thing we want to find and/or reinforce in our AGI’s world-model is not quite that. Instead it’s concepts related to the AGI’s own actions. We want the AGI to think ""I am deceiving Bob" is a very bad thing". We don't want the AGI to think ""Alice is deceiving Bob" is a bad thing", or that the abstract concept of deception is a bad thing.

In other words, getting 3rd-person labeled data is easy but unhelpful. What we really want is 1st-person labeled data.

OK, so then we can say, forget about the YouTube videos. Instead we'll do RL, or something like it. We'll have the AGI do things, and we'll label the things it does.

That is indeed a solution. But it’s a really dangerous one! We would have to catch the AGI in the act of deception before we could label deception as bad.

The real golden ticket would be if we could somehow magically transmute third-pers...
What 2026 looks like
by Daniel Kokotajlo — 2021-08-06T16:14:49.772Z

This was written for the Vignettes Workshop.[1] The goal is to write out a detailed future history (“trajectory”) that is as realistic (to me) as I can currently manage, i.e. I’m not aware of any alternative trajectory that is similarly detailed and clearly more plausible to me. The methodology is roughly: Write a future history of 2022. Condition on it, and write a future history of 2023. Repeat for 2024, 2025, etc. (I'm posting 2022-2026 now so I can get feedback that will help me write 2027+. I intend to keep writing until the story reaches singularity/extinction/utopia/etc.)

What’s the point of doing this? Well, there are a couple of reasons:

Sometimes attempting to write down a concrete example causes you to learn things, e.g. that a possibility is more or less plausible than you thought.
Most serious conversation about the future takes place at a high level of abstraction, talking about e.g. GDP acceleration, timelines until TAI is affordable, multipolar vs. unipolar takeoff… vignettes are a neglected complementary approach worth exploring.
Most stories are written backwards. The author begins with some idea of how it will end, and arranges the story to achieve that ending. Reality, by contrast, proceeds from past to future. It isn’t trying to entertain anyone or prove a point in an argument.
Anecdotally, various people seem to have found Paul Christiano’s “tales of doom” stories helpful, and relative to typical discussions those stories are quite close to what we want. (I still think a bit more detail would be good — e.g. Paul’s stories don’t give dates, or durations, or any numbers at all really.)[2]
“I want someone to ... write a trajectory for how AI goes down, that is really specific about what the world GDP is in every one of the years from now until insane intelligence explosion. And just write down what the world is like in each of those years because I don't know how to write an internally consistent, plausible trajectory. I don't know how to write even one of those for anything except a ridiculously fast takeoff.” --Buck Shlegeris

This vignette was hard to write. To achieve the desired level of detail I had to make a bunch of stuff up, but in order to be realistic I had to constantly ask “but actually though, what would really happen in this situation?” which made it painfully obvious how little I know about the future. There are numerous points whe...
Thoughts on the Recent Haitian Assassination
by Tim Liptrot — 2021-08-06T15:00:42.358Z

Thoughts on the Recent Haitian Assassination

As many of you know, a few weeks ago the president of Haiti Jovenel Moise was assassinated. This fascinating case is like a real life sherlock homes mystery complete with faked evidence, dead ends, tons of data, and plenty of motives. I'll lay out the undisputed facts of the case, then the official narrative supported by Moise's successors. Finally, I will briefly summarize what I think really happened.

The undisputed facts:

Over the past year, the Haitian regime has faced a crisis in its monopoly on violence. The purpose of repression is to prevent people from making political exchanges with your rivals, such as opposition parties, gangs and warlords. Once you are the only political exchanger in town, you can charge monopoly prices for contract enforcement, monopoly licenses, regulation and dispute arbitration. The problem is that the Moise regime was less and less able to motivate sufficient repression, enabling both gangs and opposition parties to over competing services. This problem usually results from either a shrinking budget or union-like problems with violence contractors. Moise responded by contracting with a new gang leader/war lord named Jimmy "Barbecue" Cherizier, who supplied massacres of opposition neighborhoods.

Meanwhile, Moise attempted to protect himself from coups through rotation. He knew his appointees might try to take him out, but they would need time to build their network, reputation and violence capacity to remove him. Therefore he "rotated" his Prime Ministers quickly, every year or less. On July 5th Moise announced another rotation, removing Claude Joseph and replacing him with Ariel Henry. He did not immediately sign the appointment papers.

At 1am on July 7th, a team of 28 Colombian mercenaries entered t he presidential compound. They intimidated the local police station, impersonated DA agents, and entered the presidential palace. The local police guard contingent was not present and no palace guards were injured. Moise was on the phone with his police chief for 10 minutes calling for backup while the mercenaries searched for him. They found him, confirmed his identity, then shot him dead on the spot.

Sometime after that, a series of shootouts occurred as the mercenaries attempted to escape. Several mercenaries where shot dead, most were captured, and 5 are unaccounted

Claude Joseph ...
Scoring My AstraZeneca Predictions
by Erich_Grunewald — 2021-08-06T11:52:43.427Z
Link: https://www.erichgrunewald.com/posts/scoring-my-astra-zeneca-predictions/

After a relatively good month or so, as many Western countries have begun to relax their restrictions, COVID-19 cases and deaths are on the rise again, increasing especially in Africa, the Levant and South-East Asia.[1] Over fifty thousand people die each week, and to date over four million people have died.[2] The increase is largely due to the Delta variant, which now has a foothold on every continent that isn't Antarctica.[3]

At the same time, just over a fourth of the world population has received at least one vaccination dose; about half of that number are fully vaccinated.[4] I now see that the global vaccination rate has been declining since it peaked at the end of June.[5] This makes me somewhat concerned. I am certain that the most likely way out of this purgatory phase of COVID-19 is through mass vaccination campaigns; in Texas, for example, 8,787 people have died from the virus since early February, and of those only 43 were fully vaccinated[6]. That is why I argued against the suspension of the AstraZeneca (AZ) vaccine that many European countries announced last March, calling it among other things "stupid" and "a frustrating mess".

I think the arguments I made in that post are sound and that ultimately I was right. But I also made four predictions for the future, predictions that were less impressive. I wrote:

By July 1st 2021,

the expert consensus will be that continuing with the AstraZeneca vaccinations would have saved more lives than pausing it ⇒ 95%

a majority of the listed countries will have resumed administering the AstraZeneca vaccine ⇒ 90%.

the FDA will have approved the AstraZeneca vaccine for use in the U.S. ⇒ 95%

at least one elected official from one of the listed countries will have admitted that it was a mistake to suspend vaccinations ⇒ 70%

Let us review these one by one.

Prediction #1: Continuing Vaccinations Would Have Saved Lives, 95%

Obviously we will never know the true counterfactual here, but everything I have read indicates that this is true. The European Medicines Agency's website states:

[The AZ vaccine] offers a good level of protection against COVID-19 which is a critical need in the current pandemic. The main trials showed that the vaccine has around 60% efficacy. Most side effects are mild to moderate in severity and are gone within a few days. The European Medicines Agency decided that [the vaccine's] benefits

...
Moving Away From The GPL
by jefftk — 2021-08-06T11:50:01.828Z

When I first got into open source, I published everything as

GPL.
It seemed only fair: I make my code freely available, but in exchange
people who use it need to do the same. There is a major long-standing
hole in the GPL, however, which is that it only applies to software
that runs on your device.

For example, if my friends and I write some spiffy
software to let people sing
together over the internet, licensing under the GPL wouldn't
prevent someone from making a copy with private server-side
enhancements.

The FSF
offers the AGPL
(license
text) to close this gap: it requires you to provide the source
code even if you're only making the software available over a network.
This restores the key freedom of copyleft, the right
to the source for the software you're using.

With web and mobile apps, many things that previously ran on your
local computer now run in the cloud. With virtual desktop services
like Windows
365, this can extend even to programs that weren't originally
conceived as running over a network. So I'm struggling to see a good
case for the regular GPL: if you care about copyleft, the AGPL just
seems better, closing an important loophole.

On the other hand, as I've thought more about this I've realized that
copyleft isn't a priority for me. Yes, people releasing their changes
back would be nice, but I care much more about people being able to
use the software for anything they want. With Bucket
Brigade Singing, Whistle Synth,
and my other projects, if someone used my code to make a slick
commercial project I'd see that as a good thing: now more people can
try out these ideas. The open source versions will still exist, just
as I released them, and can still be extended by people who want to
work together in the open.

Comment via: facebook
How many parameters do self-driving-car neural nets have?
by Daniel Kokotajlo — 2021-08-06T11:24:59.471Z

Take Teslas, for example. In particular, take the neural nets driving this car here. How many parameters do they have? I just need a ballpark estimate, within one order of magnitude would be great. If not for Tesla, then for Waymo or some such.

(My google-fu is failing me, though it does turn up the stat that Tesla's hardware is capable of 144 TOPS, which sounds like 1.44x10^14 operations per second.)
What are some good pieces on civilizational decay / civilizational collapse / weakening of societal fabric? 
by Eli Tyre — 2021-08-06T07:18:10.073Z

I've often heard rationalist and EAs talk about civilizational collapse, or civilizational decay: the set of trends whereby society seems to be becoming increasingly less functional, and might at some point fail entirely.

Some examples:

In Eliezer's Local Validity as a Key to Sanity and Civilization
Anna's interview with Spencer Greenberg
A lot of the discussion around covid seems to claim or imply that the US used to be better at responding to that sort of thing.
Tweets like this one (which is not intended to be representative, but is just one that I is on top of mind)

Note that I'm not confident that all of these people are talking about precisely the same phenomenon. In fact, I'm pretty sure that they're not, but they are all looking in a similar direction.

I'd like to get more orientation on how to thinking about these, and related ideas, including read whatever analyses, or speculations, that currently exist on this topic or cluster of topics.

Does anyone have suggestions?