trentmkelly/LessWrong-43k huggingface dataset:
Weak arguments against the universal prior being malign
by X4vier — 2018-06-14T17:11:53.884Z

Paul Christiano makes the case that if we use the universal prior to make important predictions, then we will end up assigning a large amount of probability mass to hypotheses which involve intelligent agents living in alternate universes who have thus-far deliberately made the correct predictions so that they might eventually manipulate us into doing what they want us to do. Paul calls these intelligent agents 'consequentialists'.

I find ideas like this very difficult to think about clearly, but I have a strong gut-feeling that the argument is not correct. I've been unable to form a crisp formal argument against Paul's proposal, but below I list a few weak reasons why the consequentialists's probability mass in the universal prior might not be as high as Paul suggests.

Unnatural output channel: It is probably the case that in the vast majority of simple universes which ultimately spawn intelligent life, the most natural output channel is not accessible to its inhabitants. Paul gives an example of such an output channel in his post: in a cellular automata we could read data by sampling the state of the first non-zero cell. The most natural thing here would probably be to start sampling immediately from t=0. However, if the automata has simple rules and a simple starting state then it will take a very large number of time-steps before consequentialist life has had time to evolve to the point at which it can start to intentionally manipulate the output cell. As another example, take our own universe: if the 'most natural' output channel in our universe corresponds to a particular location then this probably isn't inside our light-cone right now.
Unnatural input channel: Similar to natural output channels not necessarily being accessible, often it will also be impossible for a consequentialist to discern exactly what was fed in to her universe's input channel. In the example of a cellular automata, the most natural input channel is probably the initial state. This is a problem for the automata's inhabitants because, while knowing the state of the universe at a particular time lets you predict the next state, in general it won't let you deduce exactly how old the universe is or what its initial conditions were. Another source of difficulty in recovering the data fed into your universe's input channel is t

...
Notes on a recent wave of spam
by rossry — 2018-06-14T15:39:51.090Z

I think it is likely already clear to the mods, but just to make it common knowledge: There's a spam attack currently active, which takes the form of messages that copy-paste the beginning of another upthread comment (I assume, to look like a valid comment to someone not following closely), followed by a spam link.

This seems clear-cut enough (and unlikely to be a pattern a human would follow) that it might be possible to auto-delete with code, saving our mods the need to delete manually. I can't take this project on in the immediate future, but perhaps someone reading this can.
Logical Inductor Tiling and Why it's Hard
by Diffractor — 2018-06-14T06:34:36.000Z

(Tiling result due to Sam, exposition of obstacles due to me)

#Definitions:

ϕ(x––) is a sentence with a specific number plugged in for x.

Aω is the set of all infinite action sequences. For convenience, the set of actions will be taken to be {0,1}.

U:Aω→[0,1] is a utility function mapping an infinite sequence of actions to a utility, that is continuous and computable. There are some implicit assumptions in this simple definition, which will be discussed later.

Let δ:N+→(0,1] be some efficiently computable, monotonically decreasing function that is provably greater than the modulus of continuity of the utility function. More simply, if the prefix A:t of the first t actions is always enough to pin down the utility to within ϵ (which always happens because the utility function is continuous), then δ(t)≥ϵ.

We'll be using a modification of the logical induction algorithm where all traders, for turn t have access to an advice string consisting of the sequence of past actions. This is essential for giving the traders the ability to reliably bet on the sentence that represents the utility function, with the past actions plugged in, because the sequence of actions that the logical inductor takes might not be efficiently computable, so the traders would be unable to even find the right sentence to bet on.

U(p:t−1–––––,xt:f(t),a) is the LUV corresponding to what the utility function outputs. p:t−1––––– is a string of length t−1 representing the empirical past actions,xt:f(t) represents f(t)−t+1 actions selected by algorithm X, and a represents the infinite future sequence of actions selected by the logical inductor algorithm A. LUV's are defined by definition 4.8.1 of the logical induction paper Further theorems from the paper will be referenced, so it will be helpful to keep the paper open.

However, due to utility function continuity, EtU(p:t−1–––––,xt:f(t),a)≃tEtU(p:t−1–––––,a)
Therefore, we will need to define a renormalized utility U to have a nontrivial tiling result.

U(p:t−1–––––,0,xt+1:f(t),a):=U(p:t−1–––––,at,xt+1:f(t),a)−U(p:t−1–––––,0,xt+1:f(t),a)+δ(t)2δ(t)

An analogous definition applies for 1 in place of 0. In both cases, this is a LUV that always lies in the interval [0,1], by the definition of δ(t). Utility functions are invariant up to a scale and shift, so this is just a scale and shift of the original utility function. However, it is scaled so that the change

...
Anthropics made easy?
by Stuart_Armstrong — 2018-06-14T00:56:50.555Z

tl;dr: many effective altruists and rationalists seem to have key misunderstandings of anthropic reasoning; but anthropic probability is actually easier than it seems.

True or false:

The fact we survived the cold war is evidence that the cold war was less dangerous.

I'd recommend trying to answer that question in your head before reading more.

Have you got an answer?

Or at least a guess?

Or a vague feeling?

Anyway, time's up. That statement is true - obviously, surviving is evidence of safety. What are the other options? Surviving is evidence of danger? Obviously not. Evidence of nothing at all? It seems unlikely that our survival has exactly no implications about the danger.

Well, I say "obviously", but, until a few months ago, I hadn't realised it either. And five of the seven people I asked at or around EA Global also got it wrong. So what's happening?

Formalised probabilities beat words

The problem, in my view, is that we focus on true sentence like:

If we're having this conversation, it means humanity survived, no matter how safe or dangerous the cold war was.

And this statement is indeed true. If we formalise it, it becomes: P(survival | conversation) = 1, and P(survival | conversation, cold war safe) = P(survival | conversation, cold war dangerous) = 1.

Thus our conversation screens off the danger of the cold war. And, intuitively, from the above formulation, the danger or safety of the cold war is irrelevant, so it feels like the we can't say anything about it.

I think it's similar linguistic or informal formulations that have led people astray. But for the question at the beginning of the post, we aren't asking about the probability of survival (conditional on other factors), but the probability of the cold war being safe (conditional on survival). And that's something very different:

P(cold war safe | survival) = P(cold war safe)*P(survival | cold war safe)/P(survival).

Now, P(survival | cold war safe) is greater that P(survival) by definition - that's what "safe" means - hence P(cold war safe | survival) is greater than P(cold war safe). Thus survival is positive evidence for the cold war being safe.

Note that this doesn't mean that the cold war was actually safe - it just means that the likelihood of it being safe is increased when we notice we survived.

Doing anthropic probability, proper

... ,
Counterfactual Mugging Poker Game
by Scott Garrabrant — 2018-06-13T23:34:59.360Z

Consider the following game:

Player A receives a card at random that is either High or Low. He may reveal his card if he wishes.

Player B then chooses a probability p that Player A has a high card.

Player A always loses p2 dollars. Player B loses p2 dollars if the card is low and (1−p)2 dollars if the card is high.

Note that Player B has been given a proper scoring rule, and so is incentivized to give his true probability (unless he makes some deal with player A).

You are playing this game as player A. You only play one time. You are looking at a low card. Player B is not trying to make a deal with you, and will report his true probability. Player B is very good at reasoning about you, but you are in a separate room, so Player B cannot read any tells unless you show the card. Do you show your card?

Since your card is low, if you show it to player B, you will lose nothing, and get the best possible output. However, if player B reasons that if you would show your card if it was low, then in the counterfactual world in which you got a high card, player B would know you had a high card because you refused to show. Thus, you would lose a full dollar in those counterfactual worlds.

If you choose to not reveal your card, player B would assign probability 1/2 and you would lose a quarter.

I like this variant of the counterfactual mugging because it takes the agency out of the predictor. In the standard counterfactual mugging, you might reject the hypothetical and think that the predictor is trying to trick you. Here, there is a sense in which you are creating the counterfactual mugging yourself by trying to be able to keep secrets.

Also, think about this example the next time you are tempted to say that someone would only Glomarize if they had an important secret.
On the Chatham House Rule
by Scott Garrabrant — 2018-06-13T21:41:05.057Z

I have gone to several events operating under the Chatham House Rule, and have overall found it more annoying than useful. In this post, I share why I dislike the rule, how I think it can be improved, and hopefully spark others to give ideas on how to improve it. In particular, I think the part about not revealing who was at the event should be opt in. Partially, my goal is to eventually develop a modified version that might be used at future events.

The Chatham House Rule states that "When a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be revealed."

Note that the rule looks slightly ambiguous. If I am in a small conversation, clearly I cannot share who was in that conversation, but what about the list of participants as a whole? I think if you read the rule carefully, you will see that you cannot share who was at the event. Indeed, the official Chatham House website has more explanation, and explicitly states that "the list of attendees should not be circulated beyond those participating in the meeting." However a significant number of people I have asked at Chatham House Rule events were unaware that this was part of the rule.

Keeping participant lists a secret is hard

This is by far the most annoying part of the rule, and the fact that many participants often are not aware that it is part of the rule means that the attempts by people trying to follow the rule are mostly useless. It almost feels like an information hazard to me to make more people aware of what the Chatham House Rule says, since following this part of the rule is so annoying.

I have personally violated this part of the rule many times. Once, I got an email with information on logistics for the event. Inside the email were a bunch of links to google docs with details. One of the docs said that the event was under the Chatham House Rule, without explaining what it was. I at the time did not know that this applied to the list of participants. I forwarded the email to my wife, so she would know where I was and what to pack for me. The email was sent directly to me and all the rest of the participants, so the attendees were visible.

Another time, I went to a Chatham House rule event, and when I arrived, I w...
[Math] Towards Proof Writing as a Skill In Itself
by Andrew Quinn — 2018-06-13T04:39:27.480Z

In the vein of Nate, David and TT, I'm currently reading through and working on a review of Halmos's Naïve Set Theory, from the MIRI course reading list.

My background in higher mathematics is so far two 300-level courses I have taken the past two quarters at Northwestern University:

MATH 306, Combinatorics, which was mostly calculations and light on formal proofs until the end, and
MATH 300, Introduction to Higher Mathematics, which is about... Nothing but formal proofs.

MATH-300 is intentionally limited in scope so that we only prove trivial statements, until near the very end. At first I thought this would be an easy A, and a bit of a waste of time. It turned out to be neither of those things.

Writing rigorous proofs, when you don't have a lot of practice, is always harder than you would expect, even for trivially obvious statements. A few reasons I noticed:

You need to hew very closely to definitions and "allowed actions" in terms of inference. It's easy to make an 'obvious' logical leap that fails to convince the reader at all. It truly is like learning to cross inferential distances at a snail's pace.
You need to learn the common tricks of the trade -- for example, "A if and only if B" is usually proved by proving "if A then B" and then separately proving "if B then A", making it essentially a 2 sub-proof project. "If A then B" is logically equivalent to "if not B then not A", which is sometimes much easier to prove (EDIT: although discouraged, because this makes the proof non-constructive - see comments). Et cetera, et cetera.
You need to become diligent about proofreading, especially if you're typesetting with LaTeX. A single misplaced symbol will cost you a point, because the whole point is to drill the rigor into you to proofread your damn homework.

Imagine trying to build all of that, while actually learning new concepts. It's going to take you forever.

(Personal story time, feel free to skip.) I actually started with another proof-based course, Graph Theory. I dropped it, not because my grades were at all poor, but because my homework took. So. Damn. Long. And after hours of effort, I would still lose points for very small errors in places I thought were perfect.

It was infuriating. I backed off, realized that I hadn't built my skills in the right ...
Today a Tragedy 
by Logan Riggs — 2018-06-13T01:58:05.056Z

Very early this morning, my friend died in a car wreck.

When my mother called to tell me, she told me an old friend had died, but then it took her 10 seconds to compose herself to tell me exactly who.

Old friends have called and texted me throughout the day. Some seeking to console, and others to be consoled.

But it's all a tragedy.

My friends shouldn't need consolation, my mother shouldn't need to compose herself, and my friend shouldn't have died today.

A resolve arose within me:

I will solve AI alignment, and then I will work to further AGI, bringing that day closer and closer to prevent the loss of another grandson, another nephew, another friend.

But when I tried to work today, all I could do was mourn.

Will,

I'm glad I got to see you one last time in January. I'm sorry I forgot to wish you a happy birthday this year, but I've saved two dates in my calendar for you now, so I will never forget. Thanks for being my debate partner, and thanks for being my friend.

Love you. Miss you.

Logan
Epistemological Braces
by musicmage4114 — 2018-06-12T22:01:17.651Z

When engaging in debate with an eye toward victory, that is, with the goal being to change your opponent's mind, the ideal circumstance is that each participant is able to engage on equal footing, supported by nothing but the strength of their own knowledge and arguments. However, this not always the case, and we can often find ourselves engaging in debate with someone who, by any of a number of means, possesses an advantage that bolsters them against the full force of our arguments, and may even allow them to emerge victorious even if our arguments appear stronger in isolation. These advantages serve as epistemological braces, providing a solid foundation to push off of even when an argument might not otherwise be able to stand on its own.

It is worthwhile to note that oftentimes, we do not necessarily debate for our own benefit, or even the benefit of our direct opponent(s), but for the benefit of those observing - the audience. Even if we fail to change the mind of the person debating opposite us, it is possible to change the minds of those who may be listening, and certain braces (particularly those of authority) can influence our audience just as much if not more than they can influence us.

I will note here that all of these braces exist independently of participants' weight class, and multiple braces can be present at time. Certainly, it may even be that both parties in a debate have access to such braces, though the honorable debater should do their best not to rely on them if at all possible.

This list is by no means exhaustive, but those braces noted here are fairly representative of the typical external factors that often influence the outcome of a debate.

Authority

By far the most common brace, authority comes in several different flavors:

Authority of credentials

A person with authority of credentials is what we most often think of when appealing to authority, and generally involves some kind of title denoting a certain level of education, or being a visible member of a specialized profession. However, much like with fallacious appeals to authority, these titles or professions do not necessarily equal legitimacy, hence why credentials are a brace rather than a component of weight class. This brace can be removed by arguing the strength of the credentials, though using this strategy can be dangerous given the fine line between legitimate questioning ...
LW Update 2018-06-11 – Vulcan Refactor, Karma Overhaul, Colored Links, Moderation Log
by Raemon — 2018-06-12T00:49:05.508Z

I had written about the new Strong Upvotes feature over in this post, but a) I wanted to include it in an official update post, b) we've made a number of small tweaks in the past month that I wanted to summarize.

VulcanJS Refactor

Much of our progress in the past month was under-the-hood. During the Strong Upvotes rework, we needed to make a lot of changes to the VulcanJS core voting functionality. This was making it increasingly hard to keep tabs on which changes we've made to VulcanJS.

So we have now refactored the LW codebase into 3 different repositories. You can view all of them here, and the three subsections below:

Vulcan
Vulcan-Starter
Lesswrong2

This means that people contributing to the LW Open Source will need to clone all three branches into an upper level folder. (Instructions on how to do this have been added to the Lesswrong2 readme file)

Most of the time, developers should only need to make changes to the Lesswrong2 repo, and changes to the VulcanJs repos should only be made as a last resort.

Small/Big Upvotes

This was discussed in the Strong Upvotes post. On desktop computers, you can click-and-hold on the upvote button to do a larger upvote. On mobile, click twice. (After we've had a chance to see how this has played out on both device types, we may tweak the exact implementation. This is a pretty experimental change and we appreciate all feedback about it)

Karma power has been rescaled to make sense in the small/big upvote scheme, with most longterm users have a "small" upvote power of 2, and a big upvote power ranging from 4-12.

After some period of experimentation and verifying that we've gotten the right balance of karma power and inflation, we'll be retroactively applying it to old votes to ensure that old posts don't have weirdly low karma scores. The plan here is to make all old, regular upvotes into Small Upvotes (i.e. karma of 1, 2 or 3, depending on your current karma). You can then turn old upvotes into Strong Upvotes if you think it's important.

Hover-Over for vote count

Rob Bensinger and Wei_Dai had both raised interesting-but-opposite points about the value of being able to see how many votes of each type had been cast. For now, we decided to include information on how many votes, total, a given post or comment has received, but not how many upvotes/downvotes/smallvotes/bigvotes.

My hope is that ...
Admiring the Guts of Things.
by Melkor — 2018-06-11T23:12:38.084Z

Looking into the guts of things often reveals a very important perspective. Let me elaborate what I mean, via this anecdote:

Many machine learning practitioners will make a mistake of the following form:

"I ran a K-means clustering algorithm on my data, for k = 2, and it didnt show me anything interesting. Therefore, I conclude there isn't a good 2-clustering of my data."

There's a big problem here: if these practitioners knew what k-means does, they would realize the conclusion was off. K-means only works when the Euclidean metric is meaningful on their data. But, said practitioners have never looked into the guts of this method.

This example is from a 2nd year PhD student in Machine Learning at M.I.T., studying under a competent professor. So, such mistakes are not restricted to amateurs.

I posit that the general form of this mistake is: not looking into the guts of things you use every day.

*

Examples:

(A) Brains: Most humans go about their daily life using their brain all the time, and never bother to look deeply and scientifically into its function. Lesswrong writ large makes a very solid attempt to look into the guts of how the brain works.

(B) Things: Every day objects are marvels of material science, chemistry, physics, and modern engineering. Very few people can tell you remotely how a table is built.

(D) Math: Terence Tao has a concept called: "Pre-rigor, rigor, and post-rigor". The concept is here, and it is excellent: https://terrytao.wordpress.com/career-advice/theres-more-to-mathematics-than-rigour-and-proofs/. This is a must-read for anyone interested in mathematics.

This can be restated in terms of "looking into the guts of things."

Pre-rigor is the stage a mathematician is in when they have not looked deep into the guts of the mathematics they are manipulating.
Rigor is the act of looking into the messy guts of math (often compiling a lot of analysis down to Epsilon Delta arguments, or re-deriving set theory from Zermelo Frankel Axios).
Post-rigor is the act of summarizing the guts of what they see, in a compact form that allows them to finely and precisely manipulate it's contents.

Many amateur mathematicians deal with high-level summaries of deep mathematics, in the "pre-rigorous" stage. Any student who has understood calculus but not real analysis is in this phase. These students will wave the...