trentmkelly/LessWrong-43k huggingface dataset:
My AGI Threat Model: Misaligned Model-Based RL Agent
by Steven Byrnes — 2021-03-25T13:45:11.208Z

Rohin Shah advocates

a vigorous discussion of “Threat Models”, i.e. stories for how AGI is developed, what the AGI then looks like, and then what might go catastrophically wrong.

An AGI threat model. (After

Rohin Shah’s talk

.)

Ben Garfinkel likewise wants to see a

“a picture of the risk...grounded in reality”

. Richard Ngo recently had a go at answering this call with

AGI Safety From First Principles

, which is excellent and full of valuable insights, but less specific than what I have in mind. So here’s my story, going all the way from how we’ll make AGI to why it may cause catastrophic accidents, and what to do about it.

My intended audience for this post is “people generally familiar with ML and RL, and also familiar with AGI-risk-related arguments”. (If you’re in the first category but not the second, read

Stuart Russell’s book

first.) I'll try to hyperlink jargon anyway.

My AGI development model

I assume that we’ll wind up building an AGI that looks more-or-less like this:

Why do I think this model is likely? To make a long story short:

This seems like a natural extension of some of the types of AIs that researchers are building today (cf.

MuZero

).
I think that human intelligence works more-or-less this way (see

My Computational Framework for the Brain

)

…And if so, then we can have high confidence that this is a realistic path to AGI—whereas all other paths are more uncertain.
…And moreover, this offers a second R&D path to the same destination—i.e., trying to understand how the brain’s learning algorithms work (which people in both AI/ML and neuroscience are already doing all the time anyway). That makes this destination more likely on the margin.

See also: some discussion of different development paths in

Against evolution as an analogy for how humans will build AGI

.

More details about the model

The value function is a function of the latent variables in the world-model—thus, even abstract concepts like “differentiate both sides of the equation” are assigned values. The value function is updated by the reward signals, using (I assume) some generalization of TD learning

(definition)

.
I assume that the learned components (world-model, value function, planner / actor) continue to be updated in deployment—a.k.a. online learning (

definition

). This is important for the risk model below, but seems very likely—indeed, unavoidable—to me:

Online updating of the world-model

...
The Strange Case of the Amarillo Option
by adaminoue — 2021-03-25T06:16:59.849Z
Link: https://inoue.substack.com/p/the-strange-case-of-the-amarillo

A reflection on the asymmetric information problem as it relates to my own COVID-19 vaccination in January.

The first thing I’ve ever posted to LessWrong—I hope you enjoy it and welcome all feedback!
The best frequently don't rise to the top
by Adam Zerner — 2021-03-25T06:10:20.278Z

There are a variety of YouTube channels I've come across that seem like they don't get nearly enough views.

The best example that comes to my mind right now is probably Massimo Bottura. Wikipedia does a good job of describing who he is:

Massimo Bottura is an Italian restaurateur and the chef patron of Osteria Francescana, a three-Michelin-star restaurant based in Modena, Italy which has been listed in the top 5 at The World's 50 Best Restaurants Awards since 2010 and received top ratings from L'Espresso, Gambero Rosso and the Touring Club guides.

...

Osteria Francescana was ranked The World's 2nd Best Restaurant at the S.Pellegrino World's 50 Best Restaurants Awards in 2015. In June 2016 Osteria Francescana was ranked No. 1 in The World's 50 Best Restaurants and No. 2 in 2017. The restaurant returned to rank No. 1 in The World's 50 Best Restaurants in 2018.

So... arguably the best chef in the entire world. On YouTube. Showing you how he cooks. How he thinks.

Yet his videos only get 10-20k views. That might sound like a lot on the surface, but pay attention to view counts on YouTube and you'll find random housewives consistently getting hundreds of thousands of views for their cooking videos.

I think you have to consider that some sort of failure. Market inefficiency, false negative, non-meritocratic, whatever.

Another good example is David Heinemeier Hansson. Another wildly successful and influential person. DHH is the creator of Ruby on Rails, which is perhaps the most popular web framework out there. He also is a best selling author, and founder of an incredibly successful company, Basecamp.

I feel like I didn't do him justice. He's one of the people I think of when I think about who's had an impact on me. Maybe I should have let Wikipedia introduce him like I did for Massimo.

Anyway, DHH has a YouTube channel where he lets you inside his brain and shows you how he thinks about writing software. How many views do you think those videos get? It's in the same ballpark as Massimo: 10-60k. And also similar to Massimo, you can find people who are way less qualified consistently getting hundreds of thousands of views on similar videos.

I want to focus on my core point here. DH6, not DH5. You might be able to make points about how being brilliant doesn't necessarily make you good on camera, or about how marketing and promotion matters a lot, or about releasing videos on...
Curiosity needs courage
by Amir Bolous — 2021-03-25T03:38:02.162Z

This is a linkpost for https://amirbolous.com/posts/curiosity

Introduction

Growing Up

The Missing Link

Closing Thoughts

Introduction

One of the most courageous people I've ever met is my five year old cousin Max. He has no trouble sprinting across the road when cars speed by, which is why we have to hold his hand everywhere we go. He loves being thrown very high in the air, while I get nauseous just thinking about roller coasters. And he enjoys sticking his hands (and sometime other body parts for the matter) in the hottest, coldest, and stickiest surfaces he can find - from flames to ice to spiders.

Until a couple of months ago, I mistook his courage for stupidity. At least, that's the narrative I was taught by society. "How would he know any better, he's just a kid. " Looking back at some of Max's worst best moments, that narrative didn't really add up. In fact, it felt a little unfair because it implied that "Max did something stupid because he is ignorant." The more "stupid" something a child did was, the more we attributed this to being young and ignorant. However, I realize now this framing is harmfully wrong. The way we need to be thinking about this is "Max is ignorant, so he did something stupid to learn about the world."

For example, when Max put his hand near a flame, I focused more on the 20-minute episode of distracting crying that followed, rather than the twinkle in his eyes as he reached out for the flame to figure out what the hell it was.

So Max wasn't blindly reaching for the flame. Max was reaching for the flame because he was curious. What does this feel like? What is this? What does it do?

In fact, everything that Max did, from touching a spider, to trying to run across the road was his system for figuring out the inner workings of the world. And he was fine with looking stupid because he was fearless. And more importantly, because nothing was going compromise his ability to figure out why things are the way that they are.

Growing Up

When we grow up, we develop a lot of baggage that hinders us from doing this. We become afraid of what our friends or peers will think about us if we ask something weird out loud. We become afraid of looking stupid. We settle for confusion rather than face the discomfort of trying to understand something that is confusing. It's easy to admit that you don't understand something well and even easier to convince...
Why 1-boxing doesn't imply backwards causation
by Chris_Leong — 2021-03-25T02:32:12.973Z

I have written extensively on Newcomb's problem, so I assumed that I must have written up a clear explanation for this at some point. However, this doesn't seem to be the case, so I suppose I'm doing it now. The TLDR is that only one counterfactual is factual and the rest are constructed. Since they're constructed, there's no real requirement for them to have the same past as the factual and, in fact, if we value consistency, then the most natural way to construct them will involve tweaking the past.

The Student and the Exam

I've covered this example in a past post, so feel free to skip to the next section if you know it.

Suppose a student has a test on Friday. They are considering whether to study or go to the beach. Since the universe deterministic, they conclude that the outcome is already fixed and has been fixed since the start of time. Therefore, they figure out that they may as well not bother to study. Is there anything wrong with this reasoning?

My answer is that there are two views we can take of the universe.

Raw Reality: From this perspective, we are only looking at the universe as it is in its raw form; that is the territory component of the map and territory. This means that the outcome is fixed, but that the choice of whether or not to study is fixed as well. Or to be clear, the outcome is fixed in part due to the student's decision being fixed; and not independently of it. From within this view, we can't construct anything other than a Trivial Decision Theory problem as there's only a single choice we can take.

(Update: Eliezer seems to adopt a similar view here).

Augmented Reality: This perspective is created by constructing counterfactuals. It is only from within this perspective that we can talk about the student having a choice. The fact that the student necessarily obtains a particular outcome places no limitation on the counterfactuals - which are by definition not factual and were expressly created for the purpose of considering the universe travelling down a different path.

Newcomb's Problem

One of the most confusing aspects of Newcomb's Problem, is that the one-boxing solution seems to depend on backwards causation. It seems to rely on the reasoning that if I made a different choice, this would cause Omega to make a different prediction. I'll assume determinism as if we have a free will independent of determinism, Omega wouldn't be able to be...
Write a business plan already
by bfinn — 2021-03-25T00:13:21.700Z

90% of start-ups go bust, often due to poor planning. Founders know they should write a business plan, yet many never do – spouting lame excuses based on misconceptions and cognitive biases (see footnote[1]).

Or else, they throw together a sketchy, useless plan. Or an over-optimistic one they don’t believe. Or produce a plan way too late; ignore expert feedback; or don’t actually use the plan.

Even if the start-up survives, poor planning probably wasted lots of time, money, and opportunities.

So here I explain why you should write a business plan – and why excuses for not doing so are flawed. (And my next post explains how and when to write a plan.)

A parable

A builder has an idea – no, a vision – for a skyscraper. It’s so vivid in his head – he talks about it so animatedly, so truly believes in it – that he just knows he doesn’t need an architectural plan or that kind of crap. Besides, it might take him weeks, he’s not sure he can draw, and it wouldn’t progress things anyhow. Don’t plan, act!

Admittedly, he’s more of a wannabe builder, as he’s never actually built anything before. But he’s googled an article called Build Your Very Own Skyscraper by a top property developer (D. Trump), and can figure out the rest as he goes along. Fired up with excitement and self-belief, he buys some tools and sets to work.

What are the chances his skyscraper won’t get built, or it collapses? Maybe 90%?

You, dear reader, are the builder; your start-up is the skyscraper; and here it is, built without a plan:

or, more likely, this:

What are plans for?

An architectural plan lets you examine a building design from different angles, check that all its elements make sense (e.g. elevators, lighting, plumbing, parking), and get expert opinions (from structural engineers, quantity surveyors, planning authorities) before construction begins. You can revise the design in response to feedback, or abandon it if it seems unsafe or uneconomic. This process takes time and effort – but saves far more during construction, and cuts the risk enormously.

A business plan is a simulation

Similarly, a business plan lets founders examine a start-up idea on paper, check it makes sense, run it past experts, revise and improve it, and decide whether to go ahead. It’s a simulation of a business, in words and numbers.

Writing a plan helps you flesh out your idea; clarify your assumptions; research missing infor...
How You Can Gain Self Control Without "Self-Control"
by spencerg — 2021-03-24T23:38:25.926Z

Many people can’t understand how Ty does it.

Consider a typical weekday for Ty. That means waking up at 6:30 am and getting ready to run. It's freezing, and Ty's lips turn blue, but that doesn't stop Ty from going ten miles. After that, it's time for a 12-hour workday, including two three-hour stretches with no break. Ty's meals are a salad, some hummus, some cheese, fruit, and eight no-sugar protein bars. At 9 pm, Ty decides to hit the climbing gym for an hour and finally wraps up the day with another 90 minutes of work. Ty gets in bed at 12:30 am for six hours of sleep.

This schedule, as far as Ty is concerned, is easy and unremarkable. How could this schedule possibly feel easy?

You, like most people that know Ty, probably assume Ty has remarkable self-control. But the interesting thing - that Ty will readily admit - is that Ty really doesn't. At least not the sort of "self-control" you're likely thinking about.

To understand how Ty lives such a self-controlled life, let's peel back the layers of what self-controlled behavior actually is. Rather than the typical approach of treating it as a black box, we're going to rip it apart to explore which pieces are easier to change and which ones you may be stuck with for life. By adjusting these easier-to-change parts, you may be able to increase how much control you have in your life, even if you can't change your "self-control” (as some would think of it). If you can achieve this, it may well translate into making healthier choices and achieving more of your life goals.

In this article we'll be exploring:

Nine traits of self-controlled behavior (what self-control looks like when we peel it apart)
How these traits differ from what most people think of as "self-control"
Whether self-control is genetic
Twelve strategies for gaining more control in your life
A step-by-step process for applying these strategies
The strange case of Ego Depletion (where, after hundreds of studies, scientists still disagree about whether self-control can get drained like a battery)

Before We Get To Self-Control: What's This Thought Saver thing?

In a moment you'll notice that something is different about this article. It has Thought Saver widgets embedded in it, designed to help you absorb the content as you go! Just click on each of those widgets to quiz yourself on what you've learned so far. You can also click here if you want to try ...
Magic Shoes
by onur — 2021-03-24T21:14:48.743Z

While taking their daily stroll around the lake, Alice and Bob encounter a mysterious box. There is a note written on it:

~MAGIC SHOES INSIDE~

These shoes will allow you to walk on water if you follow these 3 simple steps:

Believe that you can do it.

Say 'Aqua Esto Terra!' 3 times to activate the shoes.

Step forward towards the water without any hesitation.

Alice is excited by this encounter, because she has recently been into the Law of Attraction, and manifesting a miracle in her life. Thinking that this is a sign from the universe, she hastily grabs the box and begins to put on the shoes. They seem like normal sneakers, and nothing feels out of the ordinary.

Bob watches Alice disapprovingly as she puts on the shoes. "You'll start to believe in anything anyone tells you at this rate", he says.

Ignoring Bob, Alice approaches the waterside, getting ready to take the first step. She closes her eyes, repeats the magic words 3 times as instructed, and takes a leap of faith.

To Alice's and mostly Bob's surprise, it works. Alice feels as if standing on rigid ground as hard as clean-cut stone. It appears that the waves dampen gradually as they come closer to her feet, and level completely approximately 2 inches away from the soles. The shoes adapt to the depth of the underlying water column by moving vertically but are otherwise horizontally stable. She reaches down to see if she is standing on a platform, but her hand passes through.

While Alice takes a few steps around, Bob remains in shock. He just saw his close friend perform a miracle out of a holy book, and as a result, is experiencing a complex stream of emotions. The part of him which has been a lifelong skeptic is trying to make sense of the situation.

Alice tries to walk in different directions, to run, to jump up and down, and to tapdance. After a few minutes of experimentation, she comes back to the shore and finds that Bob has regained the ability to talk. "How... the... hell...?" he says. Alice shrugs. "Magic. I was manifesting it."

"What do you mean, magic? Let me see." He helps her get back on land, and she then takes off the shoes. Bob examines the shoes closely for a few minutes, and cries "There is nothing here to see!". He puts on the shoes himself, says the magic words 3 times, and slowly lowers his left foot into the water. However, his foot passes through the water, and he recoils due to the we...
Why is COVID reinfection rate still so uncertain.
by gleb — 2021-03-24T21:12:05.107Z

From recent observatory studies, SIREN in UK , and in Denmark, they both estimate seropositivity to give around 80% of protection agains infection. The former observed 90% protection against symptomatic cases.

How efficacious are vaccines on the seropositive population? I've only seen reports on safety analysis.
To the extend of suspecting malice, the Israel Pfizer study did not report on prior infection analysis - Study protocol states "Exclusion of patients with COVID-19 prior to the index date or matched index date."
[AN #143]: How to make embedded agents that reason probabilistically about their environments
by Rohin Shah — 2021-03-24T17:20:05.166Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Infra-Bayesianism sequence (Diffractor and Vanessa Kosoy) (summarized by Rohin): I have finally understood this sequence enough to write a summary about it, thanks to AXRP Episode 5. Think of this as a combined summary + highlight of the sequence and the podcast episode.

The central problem of embedded agency (AN #31) is that there is no clean separation between an agent and its environment: rather, the agent is embedded in its environment, and so when reasoning about the environment it is reasoning about an entity that is “bigger” than it (and in particular, an entity that contains it). We don’t have a good formalism that can account for this sort of reasoning. The standard Bayesian account requires the agent to have a space of precise hypotheses for the environment, but then the true hypothesis would also include a precise model of the agent itself, and it is usually not possible to have an agent contain a perfect model of itself.

A natural idea is to reduce the precision of hypotheses. Rather than requiring a hypothesis to assign a probability to every possible sequence of bits, we now allow the hypotheses to say “I have no clue about this aspect of this part of the environment, but I can assign probabilities to the rest of the environment”. The agent can then limit itself to hypotheses that don’t make predictions about the part of the environment that corresponds to the agent, but do make predictions about other parts of the environment.

Another way to think about it is that it allows you to start from the default of “I know nothing about the environment”, and then add in details that you do know to get an object that encodes the easily computable properties of the environment you can exploit, while not making any commitments about the rest of the environment.

Of course, so far this is just the idea of using Knightian uncertainty. The contribution of infra-Bayesianism is to show how to formally specify a decision procedure that uses Knigh...
What does the reaction to NFTs tell us about how people evaluate ecological damage?
by sxae — 2021-03-24T16:01:52.938Z

Recently, the Ethereum network finalised the ERC-721 interface standard for Non-Fungible Tokens (NFTs). This standard lays out a protocol for the exchange and ownership of this new class of assets. An extremely simple explanation of the ERC-721 standard is as follows:

A blockchain is a decentralized ledger. It can contain things called smart contracts, which are like little computer programs. A smart contract can implement the ERC-721 interface, which means that it keeps track of variables called Non-Fungible Tokens (NFT). Each token has an owner, a unique identifier, and can be traded to other people. Each token can also include metadata information about the object it is representing.

While the standard had been considered for over 2 years with initial experiments like CryptoKitties, its finalization coincided with a rapid expansion of NFT tokens and markets for trading those tokens.

And people are pissed. [1] [2] [3] [4]

If you ask people why they are pissed, they will likely tell you that it has to do with the large amount of electricity used to power global cryptocurrencies. But this is a somewhat puzzling response. Surely, if your issue is with greenhouse gasses and environmental damage, criticism should be directed towards things like using fossil fuels to generate that electricity. It seems like a general problem of production, not a specific problem of demand. The total energy usage of all computer processing on earth dwarfs the amount dedicated to mining blockchains, but I doubt that many of the people arguing that NFTs are wasteful and shouldn't exist would extend their opinion to all processing costs.

Intuitively, we might imagine that the issue here is that people perceive the use of electricity as "wastage" and therefore not worth the negative environmental impacts of other, more justified usages. However, I'm not sure that many people who propose this evaluation would support its inverse implication - that there are uses of fossil fuels to generate mains-grid electricity that are non-problematic and worth the environmental damage, given the option of renewables.

There is, in my opinion, also an element of bandwagoning. Many people have soundly placed cryptocurrency within the "bad thing" box, perhaps more have heard the "miner" lingo with its anti-environmental subtext, and so they have gathered a bunch of normative beliefs together without really f...