trentmkelly/LessWrong-43k huggingface dataset:
The Second Circle
by Zvi — 2018-05-20T13:40:00.181Z

Previously: The First Circle

Epistemic Status: One additional level down

The second Circle was at Solarium in New York City. Jacob, of the New York rationalist group, and had been getting into Circling, and decided to lead us in one to show us what it was all about.

He explained the five rules. I remember the gist – to use ‘I’ statements, talk about physical sensations in detail, to listen to other people for real, and so on – but not the exact wording he used. I do remember (mostly) the wording of rule five. Rule five was memorable. It was:

5. Everything is about furthering connection. If it would further connection, do it. If not, don’t do it.

The final rule. We’re here to win, damn it. Winning here means connection. So we give you guidelines, but don’t be a slave to them. Once you’ve faked it until you’ve made it, when the situation calls for it, throw the rules out the window.

There are two kinds of rule sets. Those that contain the final rule, and those that don’t. Games and not games.

It is very bad to include that rule where it does not belong. And also very bad to not include it, where it does belong.

We get about twenty people. Good turnout. Circle begins. Everyone is quiet.

The topic of the meetup is… circling. Circling, it seems, is about circling. We’re explicitly supposed not to talk about anything. Or try to accomplish anything, other than connect.

The art must have an end other than itself or it collapses into infinite recursion.

The infinite part takes a while. First you just go meta.

Go meta? Don’t mind if we do! Rationalists love meta.

Here we all sensed we weren’t supposed to go meta. But that meant our object level thoughts were meta thoughts. No way out.

So what talking there was, kept to the rules, but went meta.

We expressed our worry that we weren’t supposed to go meta. Which meant we had gone meta-meta. Which was even worse!

Quick! Don’t think about that!

If you are pondering what I am pondering, and I am pondering what you’re pondering, then this mutual pondering of pondering is our common experience. It builds connection!

So for a while, it felt super awkward, but also felt like it was kind of… working?

Then disaster struck – Jacob worried that disaster had struck. And felt he had to Do Something. Treat the situation as bad.

Which made it ...
The Berkeley Community & The Rest Of Us: A Response to Zvi & Benquo
by Evan_Gaensbauer — 2018-05-20T07:19:12.924Z

Background Context, And How to Read This Post

This post is inspired by and a continuation of comments I made on the post 'What is the Rationalist Berkeley Community's Culture?' by Zvi on his blog Don't Worry About the Vase. As a community organizer both online and in-person in Vancouver, Canada, my goal was to fill in what appeared to be some gaps in the conversation among rationalists mostly focused on the Berkeley community. Zvi's post was part of a broader conversation pertaining to rationalist community dynamics within Berkeley.

My commentary pertains to the dynamics between the Bay Area and other local rationality communities, informed by my own experience in Vancouver and those of rationalists elsewhere. The below should not be taken be taken as comment on rationalist community dynamics within the Bay Area. This post should be considered an off-shoot from the original conversation Zvi was contributing to. For full context, please read Zvi's original post.

I. The Rationality Community: Berkeley vs. The World

While I didn't respond to them at the time, several community members commented on Zvi's post they had similar experiences: that while some local rationality communities and their members perceive themselves in a zero-sum game with Berkeley they didn't sign up for (and, to be fair, the Berkeley community didn't consciously initiate as though it's a single agency), and some don't, a sense of what Zvi was trying to point appears ubiquitous. An example:

In my experience, the recruitment to Berkeley was very aggressive. Sometimes it felt like: “if you don’t want to move to Berkeley as soon as possible, you are not *really* rational, and then it is a waste of our time to even talk to you.” I totally understand why having more rationalists around you is awesome, but trying to move everyone into one city feels like an overkill.

Similar anecdata from local rationality communities around the world:

Melbourne.

When I met several rationalists originally from Melbourne in Berkeley a few years ago, the way they talked about the exodus of the core of the Melbourne rationality community to the Bay Area, it was a mixed assessment. Melbourne is an example of very successful local rationality community outside the Bay Area, with the usual milestones like successful EA non-profits, for-profit start-ups and rat...
Can our universe contain a perfect simulation of itself?
by Hazard — 2018-05-20T02:08:41.843Z

I was recently arguing with a friend over whether or not our universe could contain a perfect simulation of itself. I've got a feeling that this question has already been discussed on LW, and possibly discussed to the point of being answered and agreed upon. This has been a fun problem to try and think through, and I'd appreciate feedback/comments of the general form of my reasoning.

Can our universe contain a perfect simulation of itself?

It seems like what counts as a perfect simulation hides most of the complexity of this problem. For now, I'm going to hand wave a bit and say a perfect simulation is anything that can asses the truth value of a arbitrary proposition about the universe.

Simulating the universe in "hard mode" (when you are within the universe) seems to be, or at least be similar to, the problem of naturalized induction. From reading only that paper, I don't get a sense that those working on naturalized induction are trying to do anything as crazy as perfectly simulate the universe.

Creating a perfect simulation has a strong feeling of mathematical impossibility to it. Here's a chain of logic to try and find an easier proposition to disprove.

1. If you cannot create an accurate "snapshot" of the entire universe, you cannot have a system that accurately models the evolution of the universe's state over time.

2. If it is impossible for a snapshot of the universe to exist within the universe, then you can't create a snapshot.

3. If a binary string of length n called the universe, cannot be completely encoded into a sub-string of length m where m < n, then in general, a universe cannot contain a snapshot of itself.

2-3 is a jump that I think is valid, yet I'd have a hard time defending. It comes from a sense of, "If we can't solve the simplified binary string version, we won't be able to solve the harder version."

Here's a "solution" to the sub-string encoding problem that feels like cheating:

Suppose m is half the length of n . Let the sub-string be the exact same sequence as the bits from m+1 to n . Use an encoding scheme like "The sub-string tells you the bit pattern for the non sub-string part of the universe, and the sub-string bit pattern is described by the sub-string bit pattern itself."

The part that feels like screwy is the "and the... ,,,
Challenges to Christiano’s capability amplification proposal
by Eliezer Yudkowsky — 2018-05-19T18:18:55.332Z

The following is a basically unedited summary I wrote up on March 16 of my take on Paul Christiano’s AGI alignment approach (described in “

ALBA

” and “

Iterated Distillation and Amplification

”). Where Paul had comments and replies, I’ve included them below.

I see a lot of free variables with respect to what exactly Paul might have in mind. I've sometimes tried presenting Paul with my objections and then he replies in a way that locally answers some of my question but I think would make other difficulties worse. My global objection is thus something like, "I don't see any concrete setup and consistent simultaneous setting of the variables where this whole scheme works." These difficulties are not minor or technical; they appear to me quite severe. I try to walk through the details below.

It should be understood at all times that I do not claim to be able to pass Paul’s

ITT

for Paul’s view and that this is me criticizing my own, potentially straw misunderstanding of what I imagine Paul might be advocating.

Paul Christiano

Overall take: I think that these are all legitimate difficulties faced by my proposal and to a large extent I agree with Eliezer's account of those problems (though not his account of my current beliefs).

I don't understand exactly how hard Eliezer expects these problems to be; my impression is "just about as hard as solving alignment from scratch," but I don't have a clear sense of why.

To some extent we are probably disagreeing about alternatives. From my perspective, the difficulties with my approach (e.g. better understanding the forms of optimization that cause trouble, or how to avoid optimization daemons in systems about as smart as you are, or how to address X-and-only-X) are also problems for alternative alignment approaches. I think it's a mistake to think that tiling agents, or decision theory, or naturalized induction, or logical uncertainty, are going to make the situation qualitatively better for these problems, so work on those problems looks to me like procrastinating on the key difficulties. I agree with the intuition that progress on the agent foundations agenda "ought to be possible," and I agree that it will help at least a little bit with the problems Eliezer describes in this document, but overall agent foundations seems way less promising than a dir

... ,,,,,
Dimensional decoupling
by brianlui — 2018-05-19T15:26:49.996Z

This is also posted at brianlui.dog.

Cornelius has a problem in his life.

Cornelius: Emma, I know you’re really good with people. I feel silly saying this, but maybe you could help me with a problem I have?

Emma: That’s a very nice thing for you to say, Cornelius! What’s the matter?

Cornelius: I keep getting taken advantage of! A lot of my friends seem like they’re really nice, but they just take and take from me and never give anything in return. I noticed that your friends aren’t like that at all. What am I doing wrong?

Emma: I’ve met a few of your friends and I have to agree. A lot of them seem like they’re nice, but I completely expect them to take advantage of you.

Cornelius: How can you tell?

Emma: Well, it’s hard to explain in words. But I just read on my favorite blog site, brianlui.dog, about a new skill that teaches you how to understand it! Here’s how it works…

Inversion

Carl Jacobi famously recommended us to “invert, always invert.” Paul Arden wrote, “Whatever you think, think the opposite.” Peter Thiel questions whether “there’s a point where no corruption can be a bad thing.”

Inversion is a powerful mode of thinking, albeit with some drawbacks. Pre-mortems are an example of a systematic application of inversion. Charlie Munger, the right hand man to Warren Buffett and also a billionaire, is fond of quoting Jacobi’s maxim:

“Turn a situation or problem upside down. Look at it backward. What happens if all our plans go wrong? Where don't we want to go, and how do you get there? Instead of looking for success, make a list of how to fail instead - through sloth, envy, resentment, self-pity, entitlement, all the mental habits of self-defeat. Avoid these qualities and you will succeed. Tell me where I'm going to die, that is, so I don't go there.”

On the other hand, Peter Thiel was criticized for applying the inversion concept too aggressively, with headlines like “11 Head-Exploding Quotes From Peter Thiel's Bonkers New York Times Interview.” Inversion is a powerful tool, but can make us look unnecessarily contrarian if we apply it in the wrong contexts.

Coming back to Cornelius’ problem with his friends taking advantage of him, maybe Cornelius could apply the inversion principle. But this only leads to thoughts like “What if the friends who take advantage of me were actually nice?” or “What if I look for mean friends instead?” Not very useful!...
Last chance to participate in the 2018 EA Survey
by Peter Wildeford — 2018-05-19T00:37:42.751Z

This year’s EA Survey will soon be closed.

You can help the community understand itself better by taking the survey via the button below before the May 26 deadline. The survey takes about 10-20 minutes to complete.

The annual EA survey is one of the best tools we have for interpreting our actions, values, demographics, and ideas as a community. See the

most popular EA Survey post of 2017

.

Take the 2018 EA Survey

All results will be aggregated, anonymized, and made publicly available, so we can share useful knowledge among each other. Within the survey you’ll be able to opt-in to share selected information about yourself in several EA venues.

Thanks and best wishes,
Peter Hurford and the

Rethink Charity

survey team

(P.S. We apologize for reposting this on LW a second and final time, but the previous post was only available on the main page for two days and had a very small amount of completions relative to LessWrong's reach. We'd love to see more LessWrong users complete the survey, so we reposted to assure the survey is more properly balanced across our communities of interest. If you want to share the survey with others, please use this fancy share link with referral tracking:

https://www.surveymonkey.com/r/3HYW9MW.)
[CKC] [May 2018] What subjects are most important for an AI safety researcher to know? (Open Call)
by Andrew Quinn — 2018-05-18T21:11:23.799Z

[CKC]: I intend this to be a Community Knowledge Convergence project - that is to say, I do not think I am particularly well informed on the subject, and I think there's a lot to be gained from compiling an understanding of what others in the LessWrong community consider as very important, probably important, possibly important, or maybe important.

What academic subjects do you consider most important to the study of AI safety? What non-academic subjects would you recommend?

Currently, here are the subjects I think are most important.

(Non-academic) Eliezer's Sequences.
Mathematical logic.
Game theoretic economics.

You might be able to tell, I'm not very well versed in this discipline. Please comment and add more subjects for me to add to the [June 2018] posts.
Talents
by Benquo — 2018-05-18T20:30:01.179Z

For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath.

- The Gospel according to Matthew

r > g

-Thomas Piketty, Capital in the Twenty-First Century

From Jesus to Piketty, it is a commonplace that wealth is a positive feedback loop.

Under one model, differential ability to steward capital, plus compounding gains, implies that perfectly benevolent people with more money than most should keep it more often than a naive expected utility maximization would suggest. On the other hand, conquering empires also experience compounding gains; the ability to leverage force into more force implies that this is a harmful positive feedback loop.

When seeking to do good, one should often attend to the specific details of the situation and take whichever action has the outcome they most prefer. But often there isn’t a very strong case for one particular action, and we’re left in need of a general heuristic for how to allocate our resources. Using explicit expected-value calculations in those circumstances will systematically underallocate resources to good uses that are less legible, and overallocate to things that illegibly cause harm. Accordingly, we need some baseline presumption for how to allocate one’s surplus between oneself, institutions one is participating in, and the rest of the world.

The deserving rich hypothesis

Money is a tool of exchange, which can’t exist unless there are goods produced and men able to produce them. Money is the material shape of the principle that men who wish to deal with one another must deal by trade and give value for value. Money is not the tool of the moochers, who claim your product by tears, or of the looters, who take it from you by force. Money is made possible only by the men who produce. Is this what you consider evil?

When you accept money in payment for your effort, you do so only on the conviction that you will exchange it for the product of the effort of others. It is not the moochers or the looters who give value to money. Not an ocean of tears not all the guns in the world can transform those pieces of paper in your wallet into the bread you will need to survive tomorrow. Those pieces of paper, which should have been gold, are a token of honor–your claim upon the energy of the men who produce. Your wallet is your statement of hope that so

...
The potential exploitability of infinite options
by Donald Hobson — 2018-05-18T18:25:39.244Z

You would think that interacting with a fully known, deterministic system, when you have unbounded computing power should be easy, you just pick the series of actions that lead to the highest reward. In cases with finitely many end states, this is the case.

Suppose you meet a benevolent genie, who says "pick a number, any real number, and I will give you that many utils".

The genie is offering you infinitely many options, with no one option being best. So whatever option you pick, you will want to go even higher. Bounded utility agents have the same problem. If you have a maximum utility of 10, and the genie offers you any amount of utility strictly less than 2, then you are still faced with an infinite sequence of ever better moves. If the agent could also use their time for something else, then they can say 2-1/3^^^^3 and then conclude that its not worth their time to specify numbers even closer to 2.

Of course you can get infinite expected utility in the infinite case. Toss a coin so that you pick 2 with prob 1/2, you pick 4 with prob 1/4 and so on. However, as soon as the coin lands, you find that you have finite utility, given an option to flip again you always will. There is also the question of adding 1 to all utilities on the coin is good, because you are guaranteed 1 util beyond what you would've gotten otherwise, or indifferent because you expect infinite utils either way. In the first case, the probabilistic option doesn't fix the problem, agents still have an infinite list of ever better options. In the second case, the agent will be ambivalent about any finite amount of utility in many situations with unbounded potential utility.

Things can be made even worse for our agent. Suppose that we start in Purgatory, and consider being there as -1 util/day. Every day the genie offers the agent an option to go to Heaven, +10 util/day for the amount of time that they have already been in Purgatory. After this period is up, they would get Oblivion, at 0 util/day.

A naive rational agent would see that it could gain more utils by rejecting the genies offer for a bit longer, than by accepting. As such it would carry on rejecting the genie and staying in Purgatory forever. A timeless decision theorist might decide that there is no meaningful difference between the decisions faced on different days, and that "always accept" yields bet...
Fundamentals of Formalisation level 2: Basic Set Theory
by  — 2018-05-18T17:21:30.969Z

Followup to Fundamentals of Formalisation level 1: Basic Logic

Basic Set Theory

The big ideas:

Axioms of Set Theory
Set Operations

To move to the next level you need to be able to:

Explain what a set is.
Calculate the intersection, union and difference of sets.
Prove two sets are equal.
Apply basic axioms of Zermelo-Fraenkel set theory.

Why this is important:

Set theory has become entrenched as the basic language with which all mathematics can be discussed. While there are more estranged parts of set theory that will likely be irrelevant to you, a fluency in the basic materials of set theory is necessary to understand more advanced mathematics.

You can find the lesson on our course platform. Good luck!
The First Circle
by Zvi — 2018-05-18T14:40:00.359Z

Epistemic Status: Squared

The following took place at an unconference about AGI in February 2017, under Chatham House rules, so I won’t be using any identifiers for the people involved.

Something that mattered was happening.

I had just arrived in San Francisco from New York. I was having a conversation. It wasn’t a bad conversation, but it wasn’t important.

To my right on the floor was a circle of four people. They were having an important conversation. A real conversation. A sacred conversation.

This was impossible to miss. People I deeply respected spoke truth. Their truth. Important truth, about the most important question – whether and when AGI would be developed, and what we could do about that to change that date, or to ensure a good outcome. Primarily about the timeline. And that truth was quite an update from my answer, and from my model of what their answers would be.

I knew we were, by default, quite doomed. But not doomed so quickly!

They were unmistakably freaking out. Not in a superficial way. In a deep, calm, we are all doomed and we don’t know what to do about it kind of way. This freaked me out too.

They talked in a different way. Deliberate, careful. Words chosen carefully.

I did not know the generation mechanism. I did know that to disturb the goings on would be profane. So I sat on a nearby couch and listed for about an hour. I said nothing.

At that time, decision was made to move to another room. I followed, and during the walk was invited to join. Two participants left, two stayed, and I joined.

The space remained sacred. I knew it had different rules, and did my best to follow them. When I was incorrect, they explained. Use ‘I’ statements. Things about your own beliefs, your models, your feelings. Things you know to be true. Pay attention to your body, and how it is feeling, where things come from, what they are like. Report it. At one point one participant said they were freaking out. I observed I was freaking out. Someone else said they were not freaking out. I said I thought they were. The first reassured me they thought there was some possibility we’d survive. Based on their prior statements, that was an update. It helped a little.

I left exhausted by the discussion, the late hour and the three hour time zone shift, and slept on it. Was this people just now waking up, perhaps not even fully? Or we...