trentmkelly/LessWrong-43k huggingface dataset:
Topological Fixed Point Exercises
by Scott Garrabrant — 2018-11-17T01:40:06.342Z

This is one of three sets of fixed point exercises. The first post in this sequence is here, giving context.

1. (1-D Sperner's lemma) Consider a path built out of n edges as shown. Color each vertex blue or green such that the leftmost vertex is blue and the rightmost vertex is green. Show that an odd number of the edges will be bichromatic.

2. (Intermediate value theorem) The Bolzano-Weierstrass theorem states that any bounded sequence in Rn has a convergent subsequence. The intermediate value theorem states that if you have a continuous function f:[0,1]→R such that f(0)≤0 and f(1)≥0, then there exists an x∈[0,1] such that f(x)=0. Prove the intermediate value theorem. It may be helpful later on if your proof uses 1-D Sperner's lemma and the Bolzano-Weierstrass theorem

3. (1-D Brouwer fixed point theorem) Show that any continuous function f:[0,1]→[0,1] has a fixed point (i.e. a point x∈[0,1] with f(x)=x). Why is this not true for the open interval (0,1)?

4. (2-D Sperner's lemma) Consider a triangle built out of n2 smaller triangles as shown. Color each vertex red, blue, or green, such that none of the vertices on the large bottom edge are red, none of the vertices on the large left edge are green, and none of the vertices on the large right edge are blue. Show that an odd number of the small triangles will be trichromatic.

5. Color the all the points in the disk as shown. Let f be a continuous function from a closed triangle to the disk, such that the bottom edge is sent to non-red points, the left edge is sent to non-green points, and the right edge is sent to non-blue points. Show that f sends some point in the triangle to the center.

6. Show that any continuous function f from closed triangle to itself has a fixed point.

7. (2-D Brouwer fixed point theorem) Show that any continuous function from a compact convex subset of R2 to itself has a fixed point. (You may use the fact that given any closed convex subset S of Rn, the function from Rn to S which projects each point to the nearest point in S is well defined and continuous.)

8. Reflect on how non-constructive all of the above fixed-point findings are. Find a parameterized class of functions where for each t∈[0,1], ft:[...
Fixed Point Exercises
by Scott Garrabrant — 2018-11-17T01:39:50.233Z

Sometimes people ask me what math they should study in order to get into agent foundations. My first answer is that I have found the introductory class in every subfield to be helpful, but I have found the later classes to be much less helpful. My second answer is to learn enough math to understand all fixed point theorems. These two answers are actually very similar. Fixed point theorems span all across mathematics, and are central to (my way of) thinking about agent foundations.

This post is the start of a sequence on fixed point theorems. It will be followed by several posts of exercises that use and prove such theorems. While these exercises aren't directly connected to AI safety, I think they're quite useful for preparing to think about agent foundations research. Afterwards, I will discuss the core ideas in the theorems and where they've shown up in alignment research.

The math involved is not much deeper than a first course in the various subjects (logic, set theory, topology, computability theory, etc). If you don't know the terms, a bit of googling, wikipedia and math.stackexchange should easily get you most of the way. Note that the posts can be tackled in any order.

Here are some ways you can use these exercises:

You can host a local MIRIx group, and go through the exercises together. This might be useful to give a local group an affordance to work on math rather than only reading papers.
You can work on them by yourself for a while, and post questions when you get stuck. You can also post your solutions to help others, let others see an alternate way of doing a problem, or help you realize that there is a problem with your solution.
You can skip to the discussion (which has some spoilers), learn a bunch of theorems from Wikipedia, and use this as a starting point for trying to understand some MIRI papers.
You can use answering these questions as a goalpost for learning a bunch of introductory math from a large collection of different subfields.
You can show off by pointing out that some of the questions are wrong, and then I will probably fix them and thank you.

The first set of exercises is here.

Thanks to Sam Eisenstat for helping develop these exercises, Ben Pace for helping edit the sequence, and many AISFP participants for testing them and noticing errors.

Meta

Read the following.

Please use the (new) spoilers feature - the symbol &...
Is Clickbait Destroying Our General Intelligence?
by Eliezer Yudkowsky — 2018-11-16T23:06:29.506Z

(Cross-posted from Facebook.)

Now and then people have asked me if I think that other people should also avoid high school or college if they want to develop new ideas. This always felt to me like a wrong way to look at the question, but I didn't know a right one.

Recently I thought of a scary new viewpoint on that subject.

This started with a conversation with Arthur where he mentioned an idea by Yoshua Bengio about the software for general intelligence having been developed memetically. I remarked that I didn't think duplicating this culturally transmitted software would be a significant part of the problem for AGI development. (Roughly: low-fidelity software tends to be algorithmically shallow. Further discussion moved to comment below.)

But this conversation did get me thinking about the topic of culturally transmitted software that contributes to human general intelligence. That software can be an important gear even if it's an algorithmically shallow part of the overall machinery. Removing a few simple gears that are 2% of a machine's mass can reduce the machine's performance by way more than 2%. Feral children would be the case in point.

A scary question is whether it's possible to do subtler damage to the culturally transmitted software of general intelligence.

I've had the sense before that the Internet is turning our society stupider and meaner. My primary hypothesis is "The Internet is selecting harder on a larger population of ideas, and sanity falls off the selective frontier once you select hard enough."

To review, there's a general idea that strong (social) selection on a characteristic imperfectly correlated with some other metric of goodness can be bad for that metric, where weak (social) selection on that characteristic was good. If you press scientists a little for publishable work, they might do science that's of greater interest to others. If you select very harshly on publication records, the academics spend all their time worrying about publishing and real science falls by the wayside.

On my feed yesterday was an essay complaining about how the intense competition to get into Harvard is producing a monoculture of students who've lined up every single standard accomplishment and how these students don't know anything else they want to do with their lives. Gentle, soft competition ...
Dimensional regret without resets
by Vanessa Kosoy — 2018-11-16T19:22:32.551Z

TLDR: I derive a variant of the RL regret bound by Osband and Van Roy (2014), that applies to learning without resets of environments without traps. The advantage of this regret bound over those known in the literature, is that it scales with certain learning-theoretic dimensions rather than number of states and actions. My goal is building on this result to derive this type of regret bound for DRL, and, later on, other settings interesting from an AI alignment perspective.

Previously I derived a regret bound for deterministic environments that scales with prior entropy and "prediction dimension". That bound behaves as O(√1−γ) in the episodic setting but only as O(3√1−γ) in the setting without resets. Moreover, my attempts to generalize the result to stochastic environments led to bounds that are even weaker (have a lower exponent). Therefore, I decided to put that line of attack on hold, and use Osband and Van Roy's technique instead, leading to an O(√1−γ) bound in the stochastic setting without resets. This bound doesn't scale down with the entropy of the prior, but this does not seem as important as dependence on γ.

Results

Russo and Van Roy (2013) introduced the concept of "eluder dimension" for the benefit of the multi-armed bandit setting, and Osband and Van Roy extended it to a form suitable for studying reinforcement learning. We will consider the following, slightly modified version of their definition.

Given a real vector space Y, we will use PSD(Y) to denote the set of positive semidefinite bilinear forms on Y and PD(Y) to denote the set of positive definite bilinear forms on Y. Given a bilinear form B:Y×Y→R, we will slightly abuse notation by also regarding it as a linear functional B:Y⊗Y→R. Thereby, we have B(v⊗w)=B(v,w) and Bv⊗2=B(v,v). Also, if Y is finite-dimensional and B is non-degenerate, we will denote B−1:Y∗×Y∗→R the unique bilinear form which satisfies

∀v∈Y,α∈Y∗:(∀β∈Y∗:B−1(α,β)=βv)⟺(∀w∈Y:B(v,w)=αw)

Definition 1

Consider a set X, a real vector space Y, some F⊆{X→Y} and a family {Bx∈PSD(Y)}x∈X. Consider also n∈N, a sequence {xk∈X}k∈[n] and x∗∈X. x∗ is said to be (F,B)-dependant on {xk} when, for any f,~f∈F

n−1∑k=0Bxk(f(xk)−~f(xk))⊗2≤1⟹Bx∗(f(x∗)−~f(x∗))⊗2≤1

Otherwise, x∗ is said to be (F,B)-independent of {xk}.

Definition 2

Consider a set X, a real vector space Y and some F⊆{X→Y}. The Russo-Van Roy-Osband dimension (RVO dimension) dimRVOF is the

...
History of LessWrong: Some Data Graphics
by Said Achmiz — 2018-11-16T07:07:15.501Z

Some graphs showing posting activity on LessWrong through the years.

NOTE: If you’re reading this post on GreaterWrong, you can click on the images to enlarge, zoom in, and click through them all as a slideshow.

Comments per post:

The same thing, on a log scale:

Posts per month:

The 100 most prolific authors over LessWrong’s lifespan:

The same thing, on a log scale:

Whose posts have generated the most total discussion?

As above, but on a log scale:

Data available in a Google Docs spreadsheet. (Or download in CSV format.)

You can also download an Excel spreadsheet, which contains the above graphs and some intermediate processed data.

Edit 2018-11-16: Updated data; corrected some minor abnormalities caused by data retrieval issue. (If you’ve downloaded the data already, please re-download the corrected versions—the links are the same.)
Sam Harris and the Is–Ought Gap
by Tyrrell_McAllister — 2018-11-16T01:04:08.185Z

Many secular materialist are puzzled by Sam Harris's frequent assertion that science can bridge Hume's is–ought gap. Indeed, bafflement abounds on both sides whenever he debates his "bridge" with other materialists. Both sides are unable to understand how the other can fail to grasp elementary and undeniable points. This podcast conversation[1] with the physicist Sean Carroll provides a vivid yet amicable demonstration.

I believe that this mutual confusion is a consequence of two distinct but unspoken ways of thinking about idealized moral argumentation. I'll call these two ways logical and dialectical.

Roughly, logical argumentation is focused on logical proofs of statements. Dialectical argumentation is geared towards rational persuasion of agents. These two different approaches lead to very different conclusions about what kinds of statements are necessary in rigorous moral arguments. In particular, the is–ought gap is unavoidable when you take the logical point of view. But this gap evaporates when you take the dialectical point of view.[2]

I won't be arguing for one of these views over the other. My goal is rather to dissolve disagreement. I believe that properly understanding these two views will render a lot of arguments unnecessary.

Logical moral argumentation

Logical argumentation, in the sense in which I'm using the term here, is focused on finding rigorous logical proofs of moral statements. The reasoning proceeds by logical inference from premises to conclusion. The ideal model is something like a theory in mathematical logic, with all conclusions proved from a basic set of axioms using just the rules of logic.

People who undertake moral argumentation with this ideal in mind envision a theory that can express "is" statements, but which also contains an "ought" symbol. Under suitable circumstances, the theory proves "is" statements like "You are pulling the switch that diverts the trolley," and "If you pull the switch, the trolley will be diverted." But what makes the theory moral is that it can also prove "ought" statements like "You ought to pull the switch that diverts the trolley."[3]

Now, this "ought" symbol could appear in the ideal formal theory in one of only two ways: Either the "ought" symbol is an undefined symbol appear...
Embedded Agency (full-text version)
by Scott Garrabrant — 2018-11-15T19:49:29.455Z

Suppose you want to build a robot to achieve some real-world goal for you—a goal that requires the robot to learn for itself and figure out a lot of things that you don't already know.

There's a complicated engineering problem here. But there's also a problem of figuring out what it even means to build a learning agent like that. What is it to optimize realistic goals in physical environments? In broad terms, how does it work?

In this post, I’ll point to four ways we don’t currently know how it works, and four areas of active research aimed at figuring it out.

1. Embedded agents

This is Alexei, and Alexei is playing a video game.

Like most games, this game has clear input and output channels. Alexei only observes the game through the computer screen, and only manipulates the game through the controller.

The game can be thought of as a function which takes in a sequence of button presses and outputs a sequence of pixels on the screen.

Alexei is also very smart, and capable of holding the entire video game inside his mind. If Alexei has any uncertainty, it is only over empirical facts like what game he is playing, and not over logical facts like which inputs (for a given deterministic game) will yield which outputs. This means that Alexei must also store inside his mind every possible game he could be playing.

Alexei does not, however, have to think about himself. He is only optimizing the game he is playing, and not optimizing the brain he is using to think about the game. He may still choose actions based off of value of information, but this is only to help him rule out possible games he is playing, and not to change the way in which he thinks.

In fact, Alexei can treat himself as an unchanging indivisible atom. Since he doesn't exist in the environment he's thinking about, Alexei doesn't worry about whether he'll change over time, or about any subroutines he might have to run.

Notice that all the properties I talked about are partially made possible by the fact that Alexei is cleanly separated from the environment that he is optimizing.

This is Emmy. Emmy is playing real life.

Real life is not like a video game. The differences largely come from the fact that Emmy is within the environment that she is trying to optimize.

Alexei sees the universe as a function, and he optimizes by choosing inputs to that function that lead to greater reward than any of...
Switching hosting providers today, there probably will be some hiccups
by habryka — 2018-11-15T19:45:59.181Z

We are switching hosting providers today from Meteor Galaxy to Amazon AWS (mostly because they have better autoscaling and are cheaper). This will involve a bunch of DNS server changes, and will be the first time we hook up our new autoscaling system to the full brunt of LessWrong traffic, so I expect there will be some problems as we try to calibrate our load balancer and everything.

Things should be stable by tonight. Sorry for the inconvenience.
Clarifying "AI Alignment"
by paulfchristiano — 2018-11-15T14:41:57.599Z

When I say an AI A is aligned with an operator H, I mean:

A is trying to do what H wants it to do.

The “alignment problem” is the problem of building powerful AI systems that are aligned with their operators.

This is significantly narrower than some other definitions of the alignment problem, so it seems important to clarify what I mean.

In particular, this is the problem of getting your AI to try to do the right thing, not the problem of figuring out which thing is right. An aligned AI would try to figure out which thing is right, and like a human it may or may not succeed.

Analogy

Consider a human assistant who is trying their hardest to do what H wants.

I’d say this assistant is aligned with H. If we build an AI that has an analogous relationship to H, then I’d say we’ve solved the alignment problem.

“Aligned” doesn’t mean “perfect:”

They could misunderstand an instruction, or be wrong about what H wants at a particular moment in time.
They may not know everything about the world, and so fail to recognize that an action has a particular bad side effect.
They may not know everything about H’s preferences, and so fail to recognize that a particular side effect is bad.
They may build an unaligned AI (while attempting to build an aligned AI).

I use alignment as a statement about the motives of the assistant, not about their knowledge or ability. Improving their knowledge or ability will make them a better assistant — for example, an assistant who knows everything there is to know about H is less likely to be mistaken about what H wants — but it won’t make them more aligned.

(For very low capabilities it becomes hard to talk about alignment. For example, if the assistant can’t recognize or communicate with H, it may not be meaningful to ask whether they are aligned with H.)

Clarifications

The definition is intended de dicto rather than de re. An aligned A is trying to “do what H wants it to do.” Suppose A thinks that H likes apples, and so goes to the store to buy some apples, but H really prefers oranges. I’d call this behavior aligned because A is trying to do what H wants, even though the thing it is trying to do (“buy apples”) turns out not to be what H wants: the de re interpretation is false but the de dicto interpretation is true.
An aligned AI can make errors, including moral or psychological errors, and fixing those errors isn’t part of my definition of alignment exce

...
The Inspection Paradox is Everywhere
by Chris_Leong — 2018-11-15T10:55:43.654Z
Link: https://allendowney.blogspot.com/2015/08/the-inspection-paradox-is-everywhere.html?fbclid=IwAR30wr-DdUA6LXEmz4Zfl0IAv9LqUBHUATKY154THEN-SpoYSJqJqvamUuc

This post describes situations where the average of an attribute observed by a participant is larger than the average over all elements.
Explore/Exploit for Conversations
by Hazard — 2018-11-15T04:11:30.372Z

Something status: I've started to try on an explore/exploit lens for thinking about conversation, and I like it. Here's some rough ideas. On the editing pass of this, a feel like I haven't properly separated the important ideas explicitly enough, but it's fine.

You are in explore mode if you are introducing ideas/topics to the conversation and aren't sure how much the others will enjoy them. You are in exploit mode if you are talking about stuff that you already know everyone will enjoy.

Being in explore mode requires feeling "comfortable" with your group, and being in exploit mode requires having Common Knowledge (CK) of Common Interests (CI).

Examples:

Explore

In your anime club's board meeting you talk about the meta of how board meetings usualy go.
You got to a LW meetup and talk about a new beat-boxing technique you've been practicing.
You tell your grandma about this heavy metal band you're getting into.

Exploit

In your anime club's board meeting you talk about what new shows you want to watch at club meetings.
You got to a LW meetup and talk about x-risk.
You tell your grandma about how school/work/family is going.

I tried to pick examples that weren't secretly saying "One of these is way better than the other!" I also want to be clear that being in explore mode doesn't have to be super edgy or out there. It just needs to be a situation where you don't know what response people will have before you go there.

Claim: For a group (2+) of people to have good conversation, you either need to have Common Knowledge of juicy Common Interests to exploit, or you need to be comfortable spending time in explore mode.

Explore Mode

I think most people experience that it is easier to have explore mode conversation with smaller groups, and with people that they know well.

I claim that it feels comfortable to be in explore mode if every group member has:

A) Reason to believe they won't get trapped in a boring conversation. (i.e anyone can steer the conversation somewhere new)

B) Reason to believe that if they are boring people, someone will let them know in a non-hurtful way.

If A) and B) hold for the group, I'm going to say your group has "explore-trust". Trying to explore when you don't have explore-trust can lead to problems:

Someone starts ranting about something far outside the

...
Stoicism: Cautionary Advice
by VivaLaPanda — 2018-11-14T23:18:00.988Z

Stoicism, or life strategies in that sphere, are something that often gets positive attention. I’m referring to the wide variety of advice that is along the lines of “look for the silver lining” and “laugh off the small things”. At the core, the stoic mindset is the recognition that we have control over our emotional responses to things through the way we frame them, and that we can get quality of life improvements “for free” by better managing our own emotions.

I’ll say up front that I think most people would benefit from giving their emotions less control over their lives and that in general the toolset provided by a stoic mindset is a helpful one. I’ve held a pretty stoic view of my emotional life for a long time, and I think it has benefited me more than its harmed me. However, I think it’s worth pointing out several pitfalls of this thinking pattern to be aware of.

Sending the Wrong Social Signals

When something bad happens, other people expect you to express certain emotions. If you make a mistake at work, you are expected to feel bad and express that through your body language and speech. If however, you’ve developed more stoic thinking patterns and ask yourself “I made a mistake, but that’s already happened so instead of regretting I’m going to focus on what I can do to avoid that mistake in the future”, you’ll also likely have body language and speech that doesn’t communicate regret in the same way. Sometimes people will recognize that you are still aware of your mistake but are approaching it from a different angle, especially if they already know you, but don’t count on it.

Be aware of what you’re expected to express, particularly if the person doesn’t know you well, and either try and communicate that body language or be clear about why you aren’t.

The Danger of Insufficient Risk Aversion

The negative emotions associated with failure exist for a reason. Often times the value of a stoic mindset is that you experience the negative emotions for an event that you couldn’t have done anything about anyway, and so additional motivation to avoid the event isn’t helpful. However it can be easy to over-correct and take a stoic mindset towards the negative emotions associated with avoidable failures. This may even still be okay, a careful consideration of how to avoid failure in the future can be more effective than the reinforcement learning approach of negativ...