trentmkelly/LessWrong-43k huggingface dataset:
Alignment Newsletter #21
by Rohin Shah — 2018-08-27T16:20:01.406Z

Highlights

80K podcast with Katja Grace

(Katja Grace and Rob Wiblin): Rob Wiblin interviewed Katja Grace of AI Impacts about her work predicting the future of AI. My main takeaway was that there are many important questions in this space that almost no one is trying to answer, and that we haven't made a good enough attempt yet to conclude that it's too hard to do, so we should put more time into it. If you haven't seen AI Impacts' work before, you can get some of the most interesting results (at a high level) from listening to this podcast. There's a ton of detail in the podcast -- too much for me to summarize here.

My opinion: I don't currently think very much about timelines, intelligence explosions, and other questions that AI Impacts thinks about, but it seems very plausible to me that these could be extremely important. (I do think about discontinuities in progress and am very glad I read the

AI Impacts post

on the subject.) One point that the interview brings up is that there are very few (perhaps two?) full time equivalents working on predicting the future of AI, while there are many people working on technical AI safety, so the former is more neglected. I'm not sure I agree with this -- the number of full time equivalents doing technical AI alignment research seems quite small (on the order of 50 people). However, I do see many people who are trying to skill up so that they can do technical AI alignment research, and none who want to do better prediction, and that seems clearly wrong. I would guess that there are several readers of this newsletter who want to do technical AI alignment research, but who would have more impact if they worked in an adjacent area, such as prediction as at AI Impacts, or policy and strategy work, or in better tools and communication. Even though I'm well-placed to do technical research, I still think that common knowledge of research is a big enough bottleneck that I spend a lot of time on this newsletter. It seems likely that there is someone else who would do a better job than me, but who is set on technical safety research even though they wouldn't be as good. So I guess if you are still trying to figure out how to best help with AI alignment, or are about to start training up to do technical research, please do listen to this podcast and consider that alternative route, and v...
Reference Post: Formal vs. Effective Pre-Commitment
by Chris_Leong — 2018-08-27T12:04:53.268Z

Newcomb's Problem is effectively a problem about pre-commitment. Everyone agrees that if you have the opportunity to pre-commit in advance of Omega predicting you, then you ought to. The only question is what you ought to do if you either failed to do this or weren't given the opportunity to do this. LW-Style Decision Theories like TDT or UDT say that you should act as though you are pre-committed, while Casual Decision Theory says that it's too late.

Formal pre-commitments include things like rewriting your code, signing a legally binding contract, or providing assets as security. If set up correctly, they ensure that a rational agent actually keeps their end of the bargain. Of course, an irrational agent may still break their end of the bargain.

Effective pre-commitment describes any situation where an agent must (in the logical sense) necessarily perform an action in the future, even if there is no formal pre-commitment. If libertarian free will were to exist, then no one would ever be effectively pre-committed, but if the universe is deterministic, then we are effectively pre-committed to any choice that we make (quantum mechanics effectively pre-commits us to particular probability distributions, rather than individual choices, but for purposes of simplicity we will ignore this here and just assume straightforward determinism). This follows straight from the definition of determinism (more discussion about the philosophical consequences of determinism in a previous post).

One reason why this concept seems so weird is that there's absolutely no need for an agent that's effectively pre-committed to know that it is pre-committed until the exact moment when it locks in its decision. From the agent's perspective, it magically turns out to be pre-committed to whatever action it chooses, however, the truth is that the agent was always pre-committed to this action, just without knowing.

Much of the confusion about pre-commitment is about whether we should be looking at formal or effective pre-commitment. Perfect predictors only care about effective pre-commitment; for them formalities are unnecessary and possibly misleading. However, human-level agents tend to care much more about formal pre-commitments. Some people, like detectives or poker players, may be really good at reading people, but they're still nothing compared to a perfect predictor and most people aren...
Aumann’s Agreement Revisited
by agilecaveman — 2018-08-27T06:21:59.476Z

This post is about reviewing some of the slightly counter-intuitive conclusions that Baysean probability theory has on agreement between people’s probabilities about the same belief. You have probably heard something about Aumann’s agreement theorem saying that “with common priors, people’s estimates should converge upon gaining common knowledge about their estimates.” However, it’s worth noting that this holds only for case of common priors, new data in the form of estimates and when people have the same models of one another’s rationality. In case the priors are not common, arbitrary new data could create a further divergence in likelihoods. In case people have the same priors on propositions, but different priors on each other’s rationality, Aumann’s agreement doesn’t apply after sharing likelihoods. The theorem is still a theorem and obviously still holds given its assumptions, however the real question is how realistic those assumptions are. I would generally argue that a model that considers some partial mistrust of each other’s rationality is more realistic and could help explain why we don’t see Aumann’s agreement in practice.

Sample misuse of the theorem:

I will first go through simple cases building up examples of agreement / disagreement. If you are interested in just the “agreement with mistrust of rationality” result, skip to the bottom.

The first example is about people with different priors reacting to new evidence. Taken from Probability the Logic of Science:

The new information D is: ‘Mr N has gone on TV with a sensational claim that a commonly used drug is unsafe’, and three viewers, Mr A, MrB, and Mr C, see this. Their prior probabilities P(S|I) that the drug is safe are (0.9, 0.1, 0.9), respectively; i.e. initially, Mr A and Mr C were believers in the safety of the drug, Mr B a disbeliever. But they interpret the information D very differently, because they have different views about the reliability of Mr N. They all agree that, if the drug had really been proved unsafe, Mr N would be right there shouting it: that is, their probabilities P(D|SI) are (1, 1, 1); but Mr A trusts his honesty while Mr C does not. Their probabilities P(D|SI) that, if the drug is safe, Mr N would say that it is unsafe, are (0.01, 0.3, 0.99), respectively. Applying Bayes’theorem P(S|DI)= P(S|I) P(D|SI)/P(D|I),and expanding the denominator by the product and sum rules...
Using expected utility for Good(hart)
by Stuart_Armstrong — 2018-08-27T03:32:51.059Z

Goodhart's curse happens when you want to maximise an unknown or uncomputable U, and instead choose a simpler proxy V, and maximise that instead. Then the optimisation pressure on V transforms it into a worse proxy of U, possibly resulting in a very bad result from the U-perspective.

Many suggestions for dealing with this involve finding some other formalism, and avoiding expected utility maximisation entirely.

However, it seems that classical expected utility maximisation can work fine, as long as we properly account for all our uncertainty and all our knowledge.

The setup

Imagine that there are 1000+5 different variables that humans might care about; the default value of these variables is 0, and they are all non-negative.

Of these, 5 are known to be variables that humans actually care about, and would like to maximise as much as possible. Of the remaining 1000, an AI agent knows that humans care about 100 of these, and want their values to be high - but it doesn't know which 100.

The agent has a "budget" of 1000 to invest in any variables it chooses; if it invest X in a given variable v, that variable is set to √X. Thus there is a diminishing return for every variable.

Then set

V={The means of the 5 known variables}.

And we get a classic Goodhart scenario: the agent will invest 200 in each of these variables, setting them to just about 14, and ignore all the other variables.

It knows we care, but not about what

In that situation, we have not, however, incorporated all that we know into the definition of V. We know that there are a 100 other variables humans care about. So we can define V as:

V={The mean of the 100+5 variables the humans care about}.

Of course, we don't know what the 100 variables are; but we can compute the expectation of V.

In order to make the model more interesting, and introduce more complicated trade-offs, I'll designate some of the 1000 variables as "stiff" variables: these require 40 times more investment to reach the same value. The number of stiff variables varies between 1 and 99 (to ensure that there is always at least one variable the humans care about among the non-stiff variables).

Then we, or an agent, can do classical expected utility maximisation on V:

This graph plots various values against the number of stiff variables. The orange dots designate the values of the 5 known variables; the agent prio...
Zetetic explanation
by Benquo — 2018-08-27T00:12:14.076Z
Link: http://benjaminrosshoffman.com/zetetic-explanation/

There is a kind of explanation that I think ought to be a cornerstone of good pedagogy, and I don't have a good word for it. My first impulse is to call it a historical explanation, after the original, investigative sense of the term "history." But in the interests of avoiding nomenclature collision, I'm inclined to call it "zetetic explanation," after the Greek word for seeking, an explanation that embeds in itself an inquiry into the thing.

Often in "explaining" a thing, we simply tell people what words they ought to say about it, or how they ought to interface with it right now, or give them technical language for it without any connection to the ordinary means by which they navigate their lives. We can call these sorts of explanations nominal, functional, and formal.

In my high school chemistry courses, for instance, there was lots of "add X to Y and get Z" plus some formulas, and I learned how to manipulate the symbols in the formulas, but this bore no relation whatsoever to the sorts of skills used in time-travel or Robinson Crusoe stories. Overall I got the sense that chemicals were a sort of magical thing produced by a mysterious Scientific-Industrial priesthood in special temples called laboratories or factories, not things one might find outdoors.

It's only in the last year that I properly learned how one might get something as simple as copper or iron, reading David W. Anthony's The Horse, the Wheel, and Language and Vaclav Smil's Still the Iron Age, both of which contain clear and concrete summaries of the process. Richard Feynman's explanation of triboluminescence is a short example of a zetetic explanation in chemistry, and Paul Lockhart's A Mathematician's Lament bears strong similarities in the field of pure mathematics.

I'm going to work through a different example here, and then discuss this class of explanation more generally.

What is yeast? A worked example

Recently my mother noted that when, in science class, her teacher had explained how bread was made, it had been a revelation to her. I pointed out that while this explanation removed bread from the category of a pure product, to be purchased and consumed, it still placed it in the category of an industrial product requiring specialized, standardized inputs such as yeast. My mother observed that she didn'...
Theories of Pain
by mingyuan — 2018-08-26T22:05:17.172Z

Epistemic status: Exploratory. I did basically no actual research for this post so please don't take anything I say at face value.

Follow-up to: Book Review: Unlearn Your Pain, A Sarno-Hanson Synthesis

.

I've had problems with pain for basically my whole life, and over the years a lot of people have told me a lot of different things about why the pain happens and what I can do to fix it. This is my attempt to put all of those things in one place and see if I can make something coherent out of them.

This post grew out of the observation that for every chronic pain intervention I've encountered, there are hundreds of people who swear by it as a miracle cure (though it doesn't work for everyone), so each thing is probably at least gesturing in the direction of something true. But most practitioners are very attached to their own view of how pain works and view at least some of the others as crackpots, so I haven't seen anyone try to tie all of the theories together.

I start with brief explanations of all the theories I've looked into for more than five minutes, divided imperfectly into physical explanations and psychological explanations and presented roughly in chronological order of when I encountered them. Then I try to inelegantly smash them all together. (Note that this is basically the same path that Todd Hargrove followed, only he spent many years on it instead of a couple hours, so maybe just read his book instead.)

Survey of theories

Physical explanations

The medical establishment (in the United States)

In my experience, complaints of chronic joint pain lead doctors to test your blood for rheumatic diseases and maybe X-ray any particularly painful bones or joints. If they don’t find evidence of disease using these methods, they kind of throw up their hands and walk away. If there is something visibly wrong with a bone/joint, they either 1) recommend physical therapy, 2) recommend surgery, or 3) be like ‘welp, I hope it heals on its own, that'll be $1000.’ They may also prescribe strong painkillers, but that feels more like a cop-out than an actual intervention to me, since it's not striking at the root of the problem.

Plausibly there is something more useful that doctors sometimes do, but beyond ruling out insidious diseases I have really not gotten a lot of value out of going to the doctor for this problem. To be fair, t...
Unrolling social metacognition: Three levels of meta are not enough.
by Academian — 2018-08-25T12:00:00.000Z

Disclaimer: This post was written time-boxed to 2 hours because I think LessWrong can still understand and improve upon it; please don't judge me harshly for it.

Summary: I am generally dismayed that many people seem to think or assume that only three levels of social metacognition matter ("Alex knows that Bailey knows that Charlie knows X"), or otherwise seem generally averse to unrolling those levels. This post is intended to point out (1) how the higher levels systematically get distilled and chunked into smaller working memory elements through social learning, which leads to emotional tracking of phenomena at 6 levels of meta and higher, and (2) what I think this means about how to approach conflict resolution.

Epistemic status: don't take my word for it; conceptual points intended to be fairly self evident upon reflection; actual techniques not backed up by systematic empirical research and might not generalize to other humans; all content very much validated by my personal experiences with talking to people about feelings in real life.

Related Reading: Duncan Sabien on Common knowledge & Miasma; Ben Pace on The Costly Coordination Mechanism of Common Knowledge

I. Conceptual introduction, by example

Here's how higher levels of social metacognition get distilled down and represented in emotions that end up tracking them (if poorly). Each feeling in the example below will be followed by an unrolling of the actual event or events it is implicitly tracking or referring to.

Warning: reading this first section (I) will require a fair bit of symbolic reasoning/thinking, so you might find it tiring and prefer to skip to later sections. A better writing of this section would do more work in between these symbolic reasoning bits to distill things out and make them easier to digest.

Scale 1: One event, four levels of meta (yes, we're starting with four)

1.1) Alex leaves out the milk for 5 minutes

1.2) Bailey observes (1.1), and feels it was bad.

Unrolling of referents: Bailey felt that Alex leaving out the milk was bad.

1.3) Alex observes (1.2), and feels judged.

Unrolling of referents: Alex felt that Bailey felt that Alex leaving out the milk was bad.

1.4) Alex reflects on feeling judged, doesn't like it, and concludes that Bailey is "a downer".

Unrolling of referents: Alex felt it was bad that Alex felt that Bailey f...
Rationalist Community Hub in Moscow: 3 Years Retrospective
by Slava Matyukhin — 2018-08-25T00:28:36.796Z

Short summary: Moscow rationalist community is organized around a local venue called Kocherga. Kocherga hosts 40–50 rationality-adjacent events per month, teaches a few dozen people per year in CFAR-style applied rationality workshops, and has ambitious plans for further growth.

We on the verge of becoming profitable, but we need some funding to stay afloat. We're launching a Patreon page today.

This post is a long overdue report on the state of LessWrong Russia community and specifically on the Kocherga community space in Moscow. This post is also a call for donations.

Russian LessWrong community has existed for many years now, but there wasn't much info on lesswrong.com about how we're doing. The last general report by Yuu was posted in March 2013. Alexander230 made a few posts about Fallacymania which is one product, but it's far from the only interesting thing about LW Russia.

In the following text I'm going to cover:

History of LW Russia from 2013 to 2015
History of Kocherga anticafe, the rationalist community space we started in 2015
The list of Kocherga's successes and failures, as well as some tentative comparisons between Kocherga and Berkeley REACH.
Our current financial situation, which is the main reason I'm taking the time to do this write-up now.

Current state of Russian and Moscow community

Here's a short description of LW Russia and LW Moscow in its current state.

Our Russian Slack chat has 1500+ registered members, 150 weekly active members and 50 weekly posting members.

LessWrong.ru has a few hundred posts from Sequences and other sources (e.g. from SlateStarCodex) translated by volunteers. LessWrong.ru gets 800 daily unique visitors. There's also a wiki with >100 articles and a VK.com page with 13000 followers.

There are regular meetups in Saint-Petersburg (hard to measure, they had a few pivots in their approach in the last few years) and in Yekaterinburg.

Our Moscow community hub, Kocherga, which is the main topic of this post, hosts 40–50 public events per month (45 events in June). The attendance varies from 3-5 visitors for smaller events to 20–30 visitors for larger ones.

We have ties with a local skeptic society, critical thinking crowd, pop-science communicators, a local systems management school and a local transhumanist community.

In the following text I'm going to talk mostly about LW Mosco...
REACH Panel Nomination Form
by stardust — 2018-08-24T23:57:09.227Z

The Berkeley REACH is seeking nominations for members of a panel to help resolve interpersonal issues that arise in or around REACH or the communities it was created to serve. See the REACH Panel Mandate document for more details about the panel.

Please submit one nomination per qualified candidate!

Nominations are open until September 7th, 2018.
Changing main content font to Valkyrie? 
by habryka — 2018-08-24T23:05:42.367Z

(Probably not interesting to 90% of people, but would love to get input from our local typography nerds)

We've been using Warnock Pro as our default font for body-text and content-related elements for a while, and I've been quite happy with it on my Mac, but on Linux on Windows machines the font renders as a bit spindly and thin, and can be hard to read at some smaller font-sizes.

Given that we've been refactoring a bunch of our Typography anyways, it seems like the best time to maybe switch to a new font. In most typography-matters I tend to defer to Butterick's Practical Typography who has a good set of recommendations for fonts similar to Warnock, and my favorite one I've found so far is Butterick's one creation Valkyrie. So changing to that seems like a thing we might do in the next few weeks.

Stylistically, my favorite font continues to be ETBook, but that one sadly has a variety of rendering issues that make it impossible to use on Lesswrong. Valkyrie is about as good as Warnock in that respect, but renders much better on Windows and Linux machines in that respect, but I would love to hear recommendations for other fonts if people have any good ones (commercial fonts are fine, as long as they don't come with weird "amount of user" limits).
Turning Up the Heat: Insights from Tao's 'Analysis II'
by TurnTrout — 2018-08-24T17:54:54.344Z

Foreword

It's been too long - a month and a half since my last review, and about three months since Analysis I. I've been immersed in my work for CHAI, but reality doesn't grade on a curve, and I want more mathematical firepower.

On the other hand, I've been cooking up something really special, so watch this space!

Analysis II

12: Metric Spaces

Metric spaces; completeness and compactness.

Proving Completeness

It sucks, and I hate it.

13: Continuous Functions on Metric Spaces

Generalized continuity, and how it interacts with the considerations introduced in the previous chapter. Also, a terrible introduction to topology.

There's a lot I wanted to say here about topology, but I don't think my understanding is good enough to break things down - I'll have to read an actual book on the subject.

14: Uniform Convergence

Pointwise and uniform convergence, the Weierstrass M-test, and uniform approximation by polynomials.

Breaking Point

Suppose we have some sequence of functions f(n):[0,1]→R, f(n)(x):=xn, which converge pointwise to the 1-indicator function f:[0,1]→R (i.e., f(1)=1 and 0 otherwise). Clearly, each f(n) is (infinitely) differentiable; however, the limiting function f isn't differentiable at all! Basically, pointwise convergence isn't at all strong enough to stop the limit from "snapping" the continuity of its constituent functions.

Progress

As in previous posts, I mark my progression by sharing a result derived without outside help.

Already proven: ∫1−1(1−x2)Ndx≥1√N.

Definition. Let ϵ>0 and 0<δ<1. A function f:R→R is said to be an (ϵ,δ)-approximation to the identity if it obeys the following three properties:

f is compactly supported on [−1,1].
f is continuous, and ∫∞−∞f=1.
|f(x)|≤ϵ for all δ≤|x|≤1.

Lemma: For every ϵ>0 and 0<δ<1, there exists an (ϵ,δ)-approximation to the identity which is a polynomial P on [−1,1].

Proof of Exercise 14.8.2(c). Suppose c∈R,N∈N; define f(x):=c(1−x2)N for x∈[−1,1] and 0 otherwise. Clearly, f is compactly supported on [−1,1] and is continuous. We want to find c,N such that the second and third properties are satisfied. Since (1−x2)N is non-negative on [−1,1], c must be positive, as f must integrate to 1. Therefore, f is non-negative.

We want to show that |c(1−x2)N|≤ϵ for all δ≤|x|≤1. Since f is non-negative, we may simplify to (1−x2)N≤ϵc. Since the left-hand... ,