trentmkelly/LessWrong-43k huggingface dataset:
Tom Chivers, author of "The AI Does Not Hate You", is running an AMA on the EA Forum
by aarongertler — 2021-03-11T06:22:57.011Z
Link: https://forum.effectivealtruism.org/posts/Q6LY8Tmw6BFuat2BX/ama-tom-chivers-science-writer-science-editor-at-unherd

Tom's book on the rationalist community was generally well-received by community members. I enjoyed the book, and have continued to enjoy following Tom's work on COVID and many other things. Among science journalists, I find him to be an unusually clear writer, with good taste when it comes to choosing which topics are worth covering.

Tom is running an AMA on the EA Forum. I think LessWrongers are likely to ask good questions, so I'd love to see you join in!

(Given the community's recent conflict with certain journalists, this is an interesting chance to ask questions about journalism to someone who is... better at that job.)
TASP Ep 3 - Optimal Policies Tend to Seek Power
by Quinn — 2021-03-11T01:44:02.814Z
Link: https://technical-ai-safety.libsyn.com/3-optimal-policies-tend-to-seek-power

Welcome to the Technical AI Safety Podcast, the show where I interview computer scientists about their papers. This month I covered Optimal Policies Tend to Seek Power, which is closely related to Seeking Power is Often Robustly Instrumental in MDPs which is a part of the Reframing Impact sequence and was recently a part of the 2019 review.

The point of the show is to make papers more parsable, the interview features a detailed walkthrough padded on either side by discussion of where the work came from and where it's going.

I had a lot of fun doing this month's episode, a tricky paper to wrap my head around but very rewarding. Do let me know if you have trouble finding it on your favorite podcast app, thanks!

Show notes:

With Alex Turner

Feedback form

Request an episode

Optimal Policies Tend to Seek Power

by Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, Prasad Tadepalli

Abstract:

Some researchers have speculated that capable reinforcement learning agents are often incentivized to seek resources and power in pursuit of their objectives. While seeking power in order to optimize a misspecified objective, agents might be incentivized to behave in undesirable ways, including rationally preventing deactivation and correction. Others have voiced skepticism: human power-seeking instincts seem idiosyncratic, and these urges need not be present in reinforcement learning agents. We formalize a notion of power within the context of Markov decision processes. With respect to a class of neutral reward function distributions, we provide sufficient conditions for when optimal policies tend to seek power over the environment.

What Counts as Defection?

Non-Obstruction
Responses and Testimonies on EA Growth
by AppliedDivinityStudies — 2021-03-10T22:57:28.878Z

Follow up to Monday's post Why Hasn't Effective Altruism Grown Since 2015?. See discussions on r/scc, LessWrong and EA Forum.

I'm honored to have received responses from Scott Alexander, Katja Grace (AI Impacts), Peter Hurford (Rethink Priorities), and Rob Bensinger (MIRI), among many other insightful replies.

This is a long post, so I'm going to put the summary first:

EA has shifted from "earning to give" to "talent-focus", leading to less of as mass movement.

By some metrics, EA is continuing to grow. Most notably, non-Open Phil donations to Give Well causes are way up the last few years.

Good Ventures is intentionally holding off increased giving while Give Well and Open Philanthropy build capacity.

In short: EA is stagnating in some ways, and not in others. Overall, this seems to be according to plan, and is not a failure of growth funding to achieve its stated cause.

EA has shifted from "money-focus" to "talent-focus", and is bottlenecked by research debt

Scott Alexander provides a useful history: Around 2014, Good Ventures stepped in and started providing a huge amount of money. For comparison, Giving What We Can has recorded $222 million in donations ever, Good Ventures gives around that much every year. This made "earning to give" much less compelling. [1]

Second, as EA has grown, it's become harder and harder to rise quickly and get a "good" job at a top institution:

there's a general sense that most things have been explored, there are rules and institutions, and it's more of a problem of learning an existing field and breaking into an existing social network rather than being part of a project of building something new.

These are both compelling to me, but we shouldn't be fatalistic about the latter and just accept that intellectual movements have limited capacity. Effective altruism is aware of the problem, or at least of something similar From MIRI:

Imagine that you have been tasked with moving a cube of solid iron that is one meter on a side. Given that such a cube weighs ~16000 pounds, and that an average human can lift ~100 pounds, a naïve estimation tells you that you can solve this problem with ~150 willing friends.

But of course, a meter cube can fit at most something like 10 people around it. It doesn't matter if you have the theoretical power to move the cube if you can't bring that power to bear in an effective manner. The problem is

...
Open Problems with Myopia
by Mark Xu — 2021-03-10T18:38:09.459Z

Thanks to Noa Nabeshima for helpful discussion and comments.

Introduction

Certain types of myopic agents represent a possible way to construct safe AGI. We call agents with a time discount rate of zero time-limited myopic, a particular instance of the broader class of myopic agents. A prototypical example is a time-limited myopic imitative agent. In theory, such an agent has some desirable safety properties because a human would only take safe actions (although any imperfect imitation would be unsafe). Since the agent is time-limited myopic, it will never imitate poorly now to make it easier to imitate easier later. For example, it would never give a human a simple plan so it could more easily imitate the human executing the plan.

We might run into issues if the agent intends to myopically imitate humans but guesses incorrectly. Such an agent might witness a human purchasing paperclips, infer that humans tend to acquire paperclips, and proceed to convert the universe into paperclips. This agent would not be safe because it is not robustly capable. Myopia does not contribute to capability robustness; we only hope it helps create intent aligned agents.

In particular, SGD might produce deceptively aligned agents. One way of viewing deception is as sacrificing reward now for reward later, which suggests that time-limited myopia should prevent it. However, there are several ways time-limited myopia fails to rule out deceptive alignment.

What we mean by myopia is myopic cognition, which is distinct from myopic training. Myopic training might produce myopic cognition, but it is not sufficient. It is currently unclear precisely what myopic cognition is. We hope a proper characterization of myopic cognition will resolve the problems presented.

Following Utility ≠ Reward, we use the term “reward” for the thing given to the agent by the training environment and the term “utility” for the thing that agent is internally trying to maximize.

Open Problems

We present a useful toy environment for reasoning about deceptive alignment. In this environment, there is a button. Agents have two actions: to press the button or to refrain. If the agent presses the button, they get +1 reward for this episode and -10 reward next episode. One might note a similarity with the traditional marshmallow test of delayed gratification.

In some sense, agents that do not press the button will engage ...
[AN #141]: The case for practicing alignment work on GPT-3 and other large models
by Rohin Shah — 2021-03-10T18:30:04.004Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

The case for aligning narrowly superhuman models (Ajeya Cotra) (summarized by Rohin): One argument against work on AI safety is that it is hard to do good work without feedback loops. So how could we get feedback loops? The most obvious approach is to actually try to align strong models right now, in order to get practice with aligning models in the future. This post fleshes out what such an approach might look like. Note that I will not be covering all of the points mentioned in the post; if you find yourself skeptical, you may want to read the full post as your question might be answered there.

The author specifically suggests that we work on aligning narrowly superhuman models to make them more useful. Aligning a model roughly means harnessing the full capabilities of the model and orienting these full capabilities towards helping humans. For example, GPT-3 presumably “knows” a lot about medicine and health. How can we get GPT-3 to apply this knowledge as best as possible to be maximally useful in answering user questions about health?

Narrowly superhuman means that the model has more knowledge or “latent capability” than either its overseers or its users. In the example above, GPT-3 almost certainly has more medical knowledge than laypeople, so it is at least narrowly superhuman at “giving medical advice” relative to laypeople. (It might even be so relative to doctors, given how broad its knowledge is.)

Learning to Summarize with Human Feedback (AN #116) is a good example of what this could look like: that paper attempted to “bring out” GPT-3’s latent capability to write summaries, and outperformed the reference summaries written by humans. This sort of work will be needed for any new powerful model we train, and so it has a lot of potential for growing the field of people concerned about long-term risk.

Note that the focus here is on aligning existing ...
[Link] Whittlestone et al., The Societal Implications of Deep Reinforcement Learning
by Aryeh Englander — 2021-03-10T18:13:25.520Z
Link: https://jair.org/index.php/jair/article/view/12360/26667

While much of the information in this paper will already be familiar to many here, I still want to highlight this paper. It's a really good, clearly articulated summary of some of the key challenges facing real-world applications of advanced AI algorithms, applied to DRL in particular. The focus on DRL will make this paper particularly appealing and accessible to ML engineers and to relevant policy makers. To me it kind of feels like an updated version of the Concrete Problems in AI Safety paper, although with a bit less technical detail and with the inclusion of some discussion related more to policy and ethics. This makes the paper important if only as a reference for talking to ML engineers or policy makers who are not necessarily familiar with safety issues and/or who are not particularly concerned about longer-term issues.
Extended Picture Theory or Models inside Models inside Models
by Chris_Leong — 2021-03-10T13:24:41.197Z

This post offers a model of meaning (and hence truth) based on extending Wittgenstein's Picture Theory. I believe that this model is valuable enough to be worth presenting on its own. I've left justification for another post, but all I'll say for now is that I believe this model to be useful even if it isn't ultimately true. I'll hint at the applications to AI safety, but these won't be fully developed either.

Finally, post only covers propositions about the world, not about logical or mathematics.

Suppose we have a model of the world:

What does this mean? What does it mean for it to be true? Is this even a question that can be answered? Well, when I introspect I seem to believe that this model claims to correspond to reality and that it is true if it in fact does correspond to reality.

So, for example, that the grass really is green and that the sky really is blue.

However, this begs the question, what does "correspond" mean here? In other words, we've just discovered that I have a meta-model that my base model corresponds to reality and we're asking what it means:

A good question to ask is: What would it mean for this model to be wrong? Let's suppose we begin with a base model of a particular shape being a dodecahedron which we produced via guessing and we then examine reality which produces a model of the shape actually being a decagon. So according to our meta-model, our base model doesn't match reality.

However, suppose the shape really is a dodecahedron, but we miscounted. Then our meta-model of the relation would be wrong as it doesn't correspond to the actual relation. Or in visual terms, the correspondence that failed to hold was:

Conversely, our meta-model would be correct if this relation had held.

Naturally, we could continue climbing up levels by asking what it means for our meta-model to correspond to reality and then for our meta-meta model to correspond to reality and so on without end.

Perhaps this isn't a very satisfying answer as this chain never terminates, but perhaps this is the only kind of answer we can give.

An Expression of the Lesson

We start out with a model.

We want to know what it means for the model to correspond to reality.

Of course, we are trapped within our models. There is no way of escaping them.

We want to tunnel through into reality and discover what our model really means.

But this is impossible.

All we can do is to form a new model of...
What is the low hanging fruit of things we could be doing to improve society?
by davidgasquez — 2021-03-10T13:03:01.562Z

I was recently listening to Robin Hanson on Signaling and Self-Deception and he mentions that most of the Social Sciences discoveries haven't been applied back to society.

It wasn’t until a little while I realized, “The reason why it’s so easy to find big improvements in social science is we almost never actually apply them. We don’t actually make the improvements that we could.”

He then goes to mention Prediction Markets as an example. They also came up as a great example in a recent LW question, What are some real life Inadequate Equilibria?

I'm curious about what things we could be doing as a society that have been proved benefitial and we're not doing so far. Either because we're stuck in a bad equilibrium or they haven't reached the general public.
Quarantine variety
by KatjaGrace — 2021-03-10T08:40:11.970Z

Among people sheltering from covid, I think there is a common thought that being stuck in your home for a year begets a certain sameyness, that it will be nice to be done with.

It’s interesting to me to remember that big chunk of the variety that is missing in life comes from regular encounters with other people, and their mind-blowing tendencies to do and think differently to me, and jump to different conclusions, and not even know what I’m talking about when I mention the most basic of basic assumptions.

And to remember that many of those people are stuck in similar houses, similarly wishing for variety, but being somewhat tired of a whole different set of behaviors and thoughts and framings and assumptions.

Which means that the variety is not fully out of safe reach in the way that, say, a big lick-a-stranger party might be. At least some of it is just informationally inaccessible, like finding the correct answer to a hard math problem. If I could somehow spend a day living like a person stuck in their house across the street lives, I would see all kinds of new things. My home itself—especially with its connection to the internet and Amazon—is capable of vastly more variety than I typically see.
Even Inflationary Currencies Should Have Fixed Total Supply
by MikkW — 2021-03-10T05:41:08.883Z

Imagine you walk into a bakery, and you ask for a dozen donuts, only to receive 11 donuts. You complain that you asked for 12 donuts and only received 11, but the baker looks at you and says “No, lad. These days a dozen is only 11. You asked for a dozen and I gave you a dozen”. Or you go to a car salesman, and he tells you about a 275 horsepower vehicle he has on offer - that sounds fast! - but when you give it a test drive, it putters along at a snail’s pace. You’d be confused, right? But then he looks you in the eye, and says “Well, but horsepower doesn’t mean what it meant back in the day. Haven’t you been keeping up with the standards published by the government?”

In theory, we could define units to mean whatever we want, and we could make our units have different values from year to year. But in general, we don’t do that, because doing so is stupid - we use units to help us understand and quantify what’s going on, and changing the definitions of units just makes things confusing and makes it harder to keep track of the world. But the one measure which we still do this with is value: one of the functions of a currency is as a unit of value, but the amount of value represented by a dollar today isn’t the same as it was 50 years ago, or what it will be 50 years from now. Economically literate people are already good at adjusting for inflation, which is a hack to get around this, but people do make mistakes failing or forgetting to adjust for inflation, and an ideal unit of value shouldn’t be changing constantly over time.

Note that I’m not saying a currency shouldn’t be inflationary - there are good reasons for currencies to be inflationary (this is particularly true for UBI coins, but also in general) which I won’t get into here, but inflation can be achieved without tampering with the unit of measurement of value - instead of increasing the supply of the coin, you can make user’s accounts decay, so if someone is holding 100 currency at one point in time, they may hold 97 currency two years later, assuming that the account is left untouched. The side effects of doing this are the same as if inflation is implemented in the usual way, but leaves the unit of value untouched. [1]

There are two ways to approach the stabilization of the measurement of value: Either you can hold the total supply of currency fixed, or you can try to track a basket of goods, and main...
AXRP Episode 5 - Infra-Bayesianism with Vanessa Kosoy
by DanielFilan — 2021-03-10T04:30:10.304Z

YouTube link

This podcast is called AXRP, pronounced axe-urp and short for the AI X-risk Research Podcast. Here, I (Daniel Filan) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it’s been written and how it might reduce the risk of artificial intelligence causing an existential catastrophe: that is, permanently and drastically curtailing humanity’s future potential.

The theory of sequential decision-making has a problem: how can we deal with situations where we have some hypotheses about the environment we’re acting in, but its exact form might be outside the range of possibilities we can possibly consider? Relatedly, how do we deal with situations where the environment can simulate what we’ll do in the future, and put us in better or worse situations now depending on what we’ll do then? Today’s episode features Vanessa Kosoy talking about infra-Bayesianism, the mathematical framework she developed with Alex Appel that modifies Bayesian decision theory to succeed in these types of situations.

Before the interview, I have a quick announcement to make. In order to make this podcast better, I’ve released a survey to get feedback from you listeners. If you have a few minutes to spare, I’d greatly appreciate it if you could fill it out - you can access it here. Now, on to the main show.

Daniel Filan:
Hello everybody. Today I’m going to be talking to Vanessa Kosoy. She is a research associate at the Machine Intelligence Research Institute. She’s worked for over 15 years in software engineering. About five years ago, she started AI alignment research, and is now doing that full-time. She’s authored three papers, but today we’re going to be talking about her sequence of posts on infra-Bayesianism that was co-authored by Alex Appel. So, Vanessa, welcome to AXRP.

Vanessa Kosoy:
Thank you, Daniel.

Daniel Filan:
All right. I guess the first question is, in a nutshell, what is infra-Bayesianism?

Vanessa Kosoy:
Infra-Bayesianism is a mathematical framework that is meant to deal with the problem of non-realizability in reinforcement learning, or in the theory of reinforcement learning. The problem of non-realizability is that you have a certain hypothesis space that your algorithm is trying to learn. It is trying to learn which hypothesis is correct, but the real world is described by none of those hypotheses because i...
The Cost of a Sixth Seat
by jefftk — 2021-03-10T03:20:02.570Z

When I posted

about cars,

several
people

suggested
we might want to get something with more than five seats so we'd have
room for friends. This would make a lot of sense, except that it's
surprisingly expensive.

The cheapest new (2021) car you can get in the US right now, as far as
I can tell, is the 4-seater Chevy Spark, $13,400
MSRP in its cheapest configuration. The cheapest 5-seater is the Mitsubishi
Mirage, at $14,295. The cheapest 7-seater is the VW Tiguan, at
$25,245.

(What about 6-seaters? Sedans seating three in the front and three
in the back used to be common, but the last one in the US was the 2013 Chevy
Impala.)

Why do you pay $11k (+77%) to seat more than five? Sure, most people
don't need a car that big, but there are still a lot of
price-sensitive families.

This is not how it used to be: it was once common for sedans to offer
a station wagon variant with a third row. They were more expensive,
but not by that much. In the US this disappeared in the mid 90s; the
8-seater Buick
Roadmaster Wagon was discontinued in 1996, as was the 7-seater Toyota Camry
Wagon.

The full-size wagons disappeared because people were switching to
minivans, and then later SUVs. Some of this was fashion, but there
was also a legal aspect: being technically "light trucks" they had less
strict emissions and fuel economy standards.

I think this explains the cost jump: what was once
a relatively minor change to a standard sedan is now a relatively
minor change to a standard SUV, but an SUV is a more expensive place
to start. For some combination of fashion and regulations, no
sedan maker is interested in producing a cheap three-row wagon
variant, though there is a small market for fancy European station
wagons that seat seven, such as the Volvo V90 at $52k
and the Mercedes
E-class Wagon at $67k.

I wonder whether there's an opening for a three-row station wagon
variant of a standard sedan starting at ~$19k?

Comment via: facebook