trentmkelly/LessWrong-43k huggingface dataset:
A sketch of a value-learning sovereign
by jessicata — 2015-12-20T21:32:45.000Z

In the previous post, I discussed
three preference frameworks for goal-directed agents. In this post, I will
discuss the value-learning sovereign in more detail.

From the Arbital article on genies:

Eliezer Yudkowsky has suggested that people only confront many important problems in value alignment when they are thinking about Sovereigns, but that at the same time, Sovereigns may be impossibly hard in practice. Yudkowsky advocates that people think about Sovereigns first and list out all the associated issues before stepping down their thinking to Genies, because thinking about Genies may result in premature pruning, while thinking about Sovereigns is more likely to generate a complete list of problems that can then be checked against particular Genie approaches to see if they have become any easier.

To this end, I think it is quite useful to discuss how to create a
value-learning sovereign, even if it is not a good idea to actually create one. I should be explicit about the fact that the concrete models in this post are almost certainly wrong (even conditioning on the fact that we have to build a value-learning sovereign); they're meant to represent the best concrete illustration of value learning that I can currently write down.

Values and ontologies

We want the AI to learn human values from human behavior. Usually, values are represented as a utility function. If the type of the world history is Ω, then a utility function over Ω is of type Ω→[0,1].
To learn U, we must first have some Ω in mind -- but what could this Ω be? There are 2 plausible candidates:

The human's ontology, ΩH. I have some way of mentally representing world states. My ontology contains concepts such as "human" and "happiness". I can express values, such as caring about human happiness, in this ontology. If the AI has a representation of ΩH, then it may be able to learn the human utility function UH:ΩH→[0,1].

The AI's ontology, ΩAI. The AI will also model the world somehow.
Probably, its model will be at least partially learned by induction. It will probably make different predictions from me, due to the fact that it might be able to discover physics that I don't know about (or otherwise model the world differently). Despite
the differences between the AI's world model and my own, it is quite likely that
my terminal values could be specified well enough in the ontology of

...
Implementing CDT with optimal predictor systems
by Vanessa Kosoy — 2015-12-20T12:58:44.000Z

We consider transparent games between bounded CDT agents ("transparent" meaning each player has a model of the other players). The agents compute the expected utility of each possible action by executing an optimal predictor of a causal counterfactual, i.e. an optimal predictor for a function that evaluates the other players and computes the utility for the selected action. Since the agents simultaneously attempt to predict each other, the optimal predictors form an optimal predictor system for the reflective system comprised by the causal counterfactuals of all agents. We show that for strict maximizers, the resulting outcome is a bounded analogue of an approximate Nash equilibrium, i.e. a strategy which is an optimal response within certain resource constraints up to an asymptotically small error. For "thermalizers" (agents that choose an action with probability proportional to 2uT), we get a similar result with expected utility Es[u] replaced by "free utility" Es[u]+TH(s). Thus, such optimal predictor systems behave like bounded counterparts of reflective oracles.

Preliminaries

The proofs for this section are given in Appendix A.

We redefine E2(ll,ϕ) and E2(ll) to be somewhat smaller proto-error spaces which nevertheless yield the same existence theorems as before. This is thanks to Lemma A.1.

Construction 1

Given ϕ∈Φ, denote E2(ll,ϕ) the set of bounded functions δ:N2→R≥0 s.t.

∀ψ∈Φ:ψ≤ϕ⟹Eλkψ[δ(k,j)]=O(ψ(k)−1)

Denote

E2(ll):=⋂ϕ∈ΦE2(ll,ϕ)

Proposition 1

If ϕ∈Φ is s.t. ∃n:liminfk→∞2−knϕ(k)=0, O(ϕ−1) is a proto-error space.

For any ϕ∈Φ, E2(ll,ϕ) is an ample proto-error space. When ϕ is non-decreasing, E2(ll,ϕ) is stable.

E2(ll) is a stable ample proto-error space.

Notation

We denote O(ϕ−1∞):=O(ϕ−1)1∞. We allow the same abuse of notation for this symbol as for usual big O notation.

For any (poly,rlog)-bischeme ^Q=(Q,rQ,σQ) we use ^σkjQ to denote UrQ(k,j)×σkjQ.

For reflective systems R=(Σ,f,μ) we write indices in Σ as superscripts rather than subscripts as before.

Given π∈ΠΣ, we denote U(π)nkj:=Uπnkj2 and πnkj(x,y):=β(ev(πnkj1;x,y)).

Results

The proofs for this section are given in Appendix B.

We start by showing that choosing an action by maximizing over optimally predicted utility is an optimal strategy in some sense.

Theorem 1

Fix a proto-error space E. Consider A a finite set, μ a word ensemble and {fa:suppμ→[0,1]}a∈A. Suppose {^Pa}a∈A are E(poly,rlog)-

...
Forecasting and recursive Inhibition within a decision cycle
by  — 2015-12-20T05:37:32.423Z

When we anticipate the future, we the opportunity to inhibit our behaviours which we anticipate will lead to counterfactual outcomes. Those of us with sufficiently low latencies in our decision cycles may recursively anticipate the consequences of counterfactuating (neologism) interventions to recursively intervene against our interventions.

This may be difficult for some. Try modelling that decision cycle as a nano-scale approximation of time travel. One relevant paradox from popular culture is the farther future paradox described in the tv cartoon called Family Guy.

Watch this clip: https://www.youtube.com/watch?v=4btAggXRB_Q

Relating the satire back to our abstraction of the decision cycle, one may ponder:

What is a satisfactory stopping rule for the far anticipation of self-referential consequence?

That is:

(1) what are the inherent harmful implications of inhibiting actions in and of themselves: stress?

(2) what are their inherent merits: self-determination?

and (3) what are the favourable and disfavourable consequences as x point into the future given y number of points of self reference at points z, a, b and c?

see no ready solution to this problem in terms of human rationality, and see no corresponding problem in artificial intelligence, where it would also apply. Given the relevance to MIRI (since CFAR doesn't seem work on open-problems in the same way)

I would like to also take this opportunity to open this as an experimental thread for the community to generate a list of ''open-problems'' in human rationality that are otherwise scattered across the community blog and wiki.
PSA: even if you don't usually read Main, there have been several worthwhile posts there recently
by Kaj_Sotala — 2015-12-19T12:34:38.870Z

A lot of people have said that they never look at Main, only Discussion. And indeed, LW's Google Analytics stats say that Main only gets one-third of the views that Discussion does.

Because of this, I thought that I'd point out that December has been an unusually lively month for Main, with several high-quality posts that you may be interested in reading out if you haven't already:

LessWrong 2.0 (Vaniver): discussion about what to do with LW in order to stop its decline. Different from previous discussions in that this time, MIRI and TrikeApps have agreed to make the changes that result from the discussion.

Why startup founders have mood swings (and why they may have uses) (AnnaSalamon and Duncan_Sabien): what the title says

Results of a One-Year Longitudinal Study of CFAR Alumni (Unnamed): CFAR has studied the impact of their workshops on people a year after taking the workshops, and have promising results.

The art of grieving well (Valentine): a beautiful and important post on the function of grief, and how to make the best out of it. A post intended for a sequence on "the sub-art of subconsciously seeking out and eliminating ugh fields and also eliminating the inclination to form them in the first place".

European Community Weekend 2016 (nino): ECW2016 is confirmed to happen!

Why CFAR? The view from 2015 (PeteMichaud): a report on what CFAR has achieved in 2015, how it has changed, and what it will do in the future.
Announcing the Signal Data Science Intensive Training Program
by JonahS — 2015-12-19T00:30:00.646Z

Note: We now have a website with up to date information here: http://signaldatascience.com/.

(This post is coauthored with Robert Cordwell.)

We’re writing to announce the inaugural run of Signal Data Science’s intensive training program.

The program will train students in the core skills needed to work as a professional data scientist:

Scraping and cleaning data

Exploring and analyzing data using statistics

Presenting findings

Interviewing

By the end of the course, you’ll will be able to start with raw data and produce analyses like the one in Bayesian Adjustment of Yelp Ratings. More to the point, you’ll understand why Jonah structured the analysis the way he did and be able to do the same yourself.

You’ll also be able to produce cool visualizations like this automatic grouping of Slate Star Codex posts by topic, as shown below.

Why data science?

Making inferences from data is fundamental to understanding the world, and there’s a growing unmet need in industry for people with the relevant skills. With good instruction and peer group, smart, motivated people can quickly develop enough proficiency to get jobs in the tech sector (starting compensation ~$115k in the San Francisco Bay Area).

Why us?

The Program

We offer inquiry-based learning (no boring lecturers or unmotivating problem sets!) and an unusually intellectually curious peer group. Far from what’s typical of college classes, our model has more in common with the Math Olympiad Summer Program, where daily lectures are interspersed with on-the-spot problems and followed by long-form problems designed to build on the lesson.

Robert Cordwell is an IMO gold medalist and educational startup veteran who’s working a Facebook data science job despite his limited, self-taught experience. He’s going to be teaching math problem solving, overall presentation skills, and how to break interviews.

Jonah Sinick is a data scientist with 13 years of experience making advanced math accessible to beginners, a PhD in math from University of Illinois, and an extensive body of published work. He’ll be teaching a comprehensive technical curriculum.

Who is this for?

If you:

Are interested in data science

Passionate about learning new things

Would benefit from a social environment with others working toward the same goal

Have the programming skills to solve s

...
[Link]Rationalization is Superior to Rationality
by  — 2015-12-18T20:30:45.667Z

Philosophy and the practice of Bayesian statistics

This is a 2012 paper by Andrew Gelman and Cosma Rohilla Shalizi on what they view as a misuse of Bayesian statistics in scientific reasoning. I found this interesting because their definition of hypothetico-deductivism closely matches up with Eliezer Yudkowsky's definition of rationalization, and their definition of inductive inference closely matches up with his definition of rationality. The definitions:

Eliezer Yudkowsky:

Rationality - Starting from evidence, and then crunching probability flows, in order to output a probable conclusion.

Rationalization - Starting from a conclusion, and then crunching probability flows, in order to output evidence apparently favoring that conclusion.

Andrew Gelman and Cosma Rohilla Shalizi:

Inductive Inference - An accretion of evidence is summarized by a posterior distribution, and scientific process is associated with the rise and fall in the posterior probabilities of various models.

Hypothetico-Deductivism - Scientists devise hypotheses, deduce implications for observations from them, and test those implications. Scientific hypotheses can be rejected (i.e., falsified), but never really established or accepted in the same way.

Now, what's interesting about the paper is that in contrast to Eliezer Yudkowsky's view they argue that rationalization (hypothetico-deductivism) is the correct analytic method, and rationality as Eliezer Yudkowsky defined it is wrong. They make the following argument:

Social-scientific data analysis is especially salient for our purposes because there is general agreement that, in this domain, all models in use are wrong – not merely falsifiable, but actually false. With enough data – and often only a fairly moderate amount – any analyst could reject any model now in use to any desired level of confidence. Model fitting is nonetheless a valuable activity, and indeed the crux of data analysis. To understand why this is so, we need to examine how models are built, fitted, used and checked, and the effects of misspecification on models.

They also argue Popper made multiple errors, but that his fundamental view is closer to correct than Kuhn's, and that correct science is about attempting to falsify hypotheses. They simply disagree with how Popper went about doing it.

Another inter...
New LW Meetup: Baltimore
by FrankAdamek — 2015-12-18T17:38:31.690Z

This summary was posted to LW Main on December 11th. The following week's summary is here.

New meetups (or meetups with a hiatus of more than a year) are happening in:

Baltimore Area Meetup: 03 January 2016 02:00PM

Irregularly scheduled Less Wrong meetups are taking place in:

San Antonio Meetup: Goal Factoring Discussion: 13 December 2015 02:00PM
São Paulo: 12 December 2015 02:00PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

MIT/Boston Secular Solstice: 11 December 2015 08:00PM
[Moscow] FallacyMania game in Kocherga club: 23 December 2015 07:40PM
Moscow social meetup: Chaos is back!: 27 December 2015 02:00PM
NH Meetup: 05 January 2016 07:00PM
NYC Solstice: 19 December 2015 05:30PM
Seattle Solstice: 19 December 2015 05:00PM
Sydney solstice: 12 December 2015 05:00PM
Washington, D.C.: Sharing Book Excerpts: 13 December 2015 03:00PM

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Canberra, Columbus, Denver, London, Madison WI, Melbourne, Moscow, Mountain View, New Hampshire, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers and a Slack channel for daily discussion and online meetups on Sunday night US time.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview posts are moved to the discussion section when the new post goes up.

Please note that for your meetup to appear in the weekly meetups feature, you need to post your meetup before the Friday before your meetup!

If you check Less Wrong irregularly, consider subscribing to one...
The Art of Lawfare and Litigation strategy
by  — 2015-12-17T14:34:03.344Z

Bertrand Russell, well aware there were health risks of smoking, defended his addiction in a videotaped interview. See if you can spot his fallacy!

Today on SBS (radio channel in Australia) I heard reporters breaking the news that Nature article reports that Cancer is largely due to choices. I was shocked by what appeared to be gross violations of cultural norms around the blaming of victims. I wanted to investigate further since science reporting is notoriously inaccurate.

The BBC reports:

Earlier this year, researchers sparked a debate after suggesting two-thirds of cancer types were down to luck rather than factors such as smoking.

The new study, in the journal Nature, used four approaches to conclude only 10-30% of cancers were down to the way the body naturally functions or "luck".

"They can't smoke and say it's bad luck if they have cancer."

-Dr Yusuf Hannun, the director of Stony Brook

-http://www.bbc.com/news/health-35111449

The BBC article is roughly concordant with the SBS report.

I've had a fairly simple relationship with cigarettes. I've smoked others' cigarettes a few times, while drinking. I bought my first cigarette to try soon after I turned of age and discarded the rest of the packet. One of my favourite memories is trying a vanilla flavoured cigar. I still feel tempted to it again whenever I smell a nice scent, or think about that moment. Though now, I regularly reject offers to go to local venues and smoke hookah. Even after my first cigarette, I felt the tug of nicotine and tobacco. Though, I'm unusually sensitive to eve the mildest addictive substances, so that doesn't suprise me in respective. What does suprise me, is that society is starting to take a ubiquitous but increasingly undeniable health issue seriously despite deep entanglement with long standing way of doing things, political ideologues, individual addictions and addiction-driven political behaviour and shareholder's pockets.

Though the truth claim of the article isn't that suprising. The dangers of smoking are publicised everywhere. Emphasis mine:

13 die every day in Victoria as a result of smoking.

Tobacco use (which includes cigarettes, cigars, pipes, snuff, chewing tobacco) is the leading preventable cause of death and illness in our country. It causes more deaths annually than those killed by AIDS, alcohol, automobile accidents, murders,

...
Should AI Be Open?
by Scott Alexander — 2015-12-17T08:25:26.000Z

I.

H.G. Wells’ 1914 sci-fi book The World Set Free did a pretty good job predicting nuclear weapons:

They did not see it until the atomic bombs burst in their fumbling hands…before the last war began it was a matter of common knowledge that a man could carry about in a handbag an amount of latent energy sufficient to wreck half a city

Wells believed the coming atomic bombs would be so deadly that we would inevitably create a utopian one-world government to prevent them from ever being used. Sorry, Wells. It was a nice thought.

But imagine that in the 1910s and 1920s, some elites had started thinking really seriously along Wellsian lines. They would worry about what might happen when the first nation – let’s say America – got the Bomb. It would be unstoppable in battle and might rule the world with an iron fist. Such a situation would be the end of human freedom and progress.

So in 1920, these elites pooled their resources and made their own Manhattan Project. Their efforts bore fruit, and they learned a lot about nuclear fission; in particular, they learned that uranium was a necessary raw material. The world’s uranium sources were few enough that a single nation or coalition could get a monopoly upon them; the specter of atomic despotism seemed more worrying than ever.

They got their physicists working overtime and discovered a new type of nuke that required no uranium at all. In fact, once you understood the principles you could build one out of parts from a Model T engine. The only downside was that if you didn’t build it exactly right, its usual failure mode was to detonate on the workbench in an uncontrolled hyper-reaction that would blow the entire hemisphere to smithereens.

And so the intellectual and financial elites declared victory – no one country could monopolize atomic weapons now – and sent step-by-step guides to building a Model T nuke to every household in the world. Within a week, both hemispheres were blown to very predictable smithereens.

II.

Some of the top names in Silicon Valley have just announced a new organization, OpenAI, dedicated to “advancing digital intelligence in the way that is most likely to benefit humanity as a whole…as broadly and evenly distributed as possible.” Co-chairs Elon Musk and Sam Altman talk to Steven Levy:

Levy: How did this come about? […

...
Rationality Reading Group: Part P: Reductionism 101
by Gram_Stone — 2015-12-17T03:03:49.172Z

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Part P: Reductionism (pp. 887-935). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

P. Reductionism 101

189. Dissolving the Question - This is where the "free will" puzzle is explicitly posed, along with criteria for what does and does not constitute a satisfying answer.

190. Wrong Questions - Where the mind cuts against reality's grain, it generates wrong questions - questions that cannot possibly be answered on their own terms, but only dissolved by understanding the cognitive algorithm that generates the perception of a question.

191. Righting a Wrong Question - When you are faced with an unanswerable question - a question to which it seems impossible to even imagine an answer - there is a simple trick which can turn the question solvable. Instead of asking, "Why do I have free will?", try asking, "Why do I think I have free will?"

192. Mind Projection Fallacy - E. T. Jaynes used the term Mind Projection Fallacy to denote the error of projecting your own mind's properties into the external world. The Mind Projection Fallacy generalizes as an error. It is in the argument over the real meaning of the word sound, and in the magazine cover of the monster carrying off a woman in the torn dress, and Kant's declaration that space by its very nature is flat, and Hume's definition of a priori ideas as those "discoverable by the mere operation of thought, without dependence on what is anywhere existent in the universe"...

193. Probability is in the Mind - Probabilities express uncertainty, and it is only agents who can be uncertain. A blank map does not correspond to a blank territory. Ignorance is in the mind.

194. The Quotation is Not the Referent - It's very easy to derive extremely wrong conclusions if you don't make a clear enough distinction between your beliefs about the world, and the world itself.

195. Qualitatively Confused - Using qualitative, binary reasoning may make it easier to confuse belief and reality; if we use probability distributions, the d...
Meetup : Washington, D.C.: Fun & Games
by RobinZ — 2015-12-16T21:29:26.681Z

Discussion article for the meetup : Washington, D.C.: Fun & Games

WHEN:
20 December 2015 03:00:00PM (-0500)

WHERE:
Reynolds Center

x-posted from list. Gathering in courtyard from 3:00pm, hard start 3:30pm - until closing (7:00 pm).

We'll be meeting to hang out, play games, and engage in fun conversation.

Upcoming meetups:

Dec. 27: Holidays & Socialization

Jan. 3: Meta Meetup

Jan. 10: Fun & Games

Discussion article for the meetup : Washington, D.C.: Fun & Games