trentmkelly/LessWrong-43k huggingface dataset:
What's the best explanation of intellectual generativity?
by Raemon — 2019-06-28T18:33:29.278Z

Lately I've found myself wanting to make the argument that intellectual generativity is very important, and that you should be very careful with subtle forces that can corrode it.

"Generativity" is the sort of word that seems to come up a lot in casual conversations in my current circle but I just went looking for a good explanatory post and couldn't find one. I'm fairly confident that someone somewhere has talked about it (not necessarily on LW).

Curious if anyone knows of good existing writing?

And if anyone wanted to write up a fresh explanation that'd be cool as well. (A possible outcome is treating the answer section here as an opportunity to write a first draft that maybe turns into a post if there's consensus the answer is good)
Systems Engineering Advancement Research Initiative
by ryan_b — 2019-06-28T17:57:54.606Z
Link: http://seari.mit.edu/publications.php

This is a linkpost for a research initiative at MIT I just discovered while following up on some earlier reading. I have linked to the Publications page to make it easiest for people to get in and start perusing.

The goal of this initiative is to improve state-of-the-art systems engineering, and in particular to be able to account for uncertainty and changing contexts during and after the design phase.

What drove me to bring it to the attention of this community is the MATE program, which stands for Multi-Attribute Tradespace Exploration. This is interesting because it consists of defining desirable qualities and then building a utility function out of them, upon which design decisions will be based.

Among the publications are a series of applied attempts at building actual utility functions for real things, then using decision theory with them.
They increase the amount of things you can trade off for by expanding what they call the "Ilities" (reliability, versatility, etc). Among these new ilities is a more advanced notion of safety.
This means there is a body of applied work which has a concept of safety, calculated into utility functions, and operated on according to decision theory.

Further updates will be made in the comments as I finish readings.
Self-confirming prophecies, and simplified Oracle designs
by Stuart_Armstrong — 2019-06-28T09:57:35.571Z

I've got a paper on two Oracle[1] designs: the counterfactual Oracle and the low bandwidth Oracle. In this post, I'll revisit these designs and simplify them, presenting them in terms of sequence prediction for an Oracle with self-confirming predictions.

Predicting y

The task of the Oracle is simple: at each time t, they will output a prediction xt, in the range [−5,5]. There will then be a subsequent observation yt. The Oracle aims to minimise the quadratic loss function l(xt,yt)=(xt−yt)2.

Because there is a self-confirming aspect to it, the yt is actually a (stochastic) function of xt (though not of xt−1 or preceding xi's). Let Yt be the random variable such that Yt(xt) describes the distribution of yt given xt. So the Oracle wants to minimise the expectation of the quadratic loss:

L(xt)=(xt−Yt(xt))2.

What is the Yt in this problem? Well, I'm going to use it to illustrate many different Oracle behaviours, so it is given by this rather convoluted diagram:

.

The red curve is the expectation of Yt, as a function of xt; it is given by f(x)=E(Yt|x=xt).

Ignoring, for the moment, the odd behaviour around 2.5, y=f(x) is a curve that starts below the y=x line, climbs above it (and so has a fixed point at x=−2) in piecewise-linear fashion, and then transforms into an inverted parabola that has another fixed point at x=4. The exact equation of this curve is not important[2]. Relevant, though, is the fact that the fixed point at x=4 is attractive, while the one at x=−2 is not.

What of the blue edging? That represents the span of the standard deviation around the expectation. For any given x, the Y(x) is a normal distribution with mean f(x) and standard deviation g(x). This g(x) is given by:

So the g(x) is zero for x less than −2.5. From there, it jumps up to 1, for −2.5<x≤1. From that point onward, it starts growing linearly, being equal to x: g(x)=x. The blue edges of the diagram above are the curves of f(x)+g(x) and f(x)−g(x): the range between plus and minus one standard deviation.

Wireheading

But what is happening around x=2.5? Well, I wanted to represent the behaviour of wireheading: finding some "cheating" output that gives maximal accuracy, through hacking the system or tricking the human. These solutions are rare, so I confined them to a tiny area around x=2.5, where the Oracle has maximal accuracy and lowest variance, because it's "hacked" the problem setup.

T

...
GreaterWrong Arbital Viewer
by clone of saturn — 2019-06-28T06:32:22.278Z
Link: https://arbital.greaterwrong.com/

You can now view Arbital through GreaterWrong: https://arbital.greaterwrong.com/

Some of Arbital's features are supported and some aren't; let me know in the comments if there's anything you're particularly missing.

Thanks to emmab for downloading the content.
Instead of "I'm anxious," try "I feel threatened"
by Holly_Elmore — 2019-06-28T05:24:52.593Z
Link: https://mhollyelmoreblog.wordpress.com/2019/06/19/instead-of-im-anxious-try-i-feel-threatened/

cw: teaching to learn

I have a long history with anxiety, and I’m pretty good at noticing when it’s happening. The problem is that I’m always anxious. Noticing anxiety doesn’t snap me out of anxiety– in fact, it often produces meta-anxiety, anxiety about feeling anxious. So I’ve tried a simple reframe lately, and I’m liking the results. Instead of noting “I’m anxious,” I say to myself “I feel threatened” or “I feel threatened by x” if I know what set me off.

Anxiety is just chronically being in a state of fight or flight, and fight or flight has a stimulus. I like Sapolsky’s thesis, which is roughly that for most animals, the stimulus is always something external, a threat to safety or status. For anxious humans, the threatening stimuli are internalized, and fight or flight is either triggered or sustained by thoughts. Anxiety is the condition of feeling threatened.

And yet, noticing that I feel threatened is much more specific than noticing that I’m anxious, whether I can identify the threat or not. It makes what I’m feeling less about me (I’m just anxious; my perception is inaccurate; oh, why don’t I just stop???) and more about the pattern of behavior (I’m reacting this way because I perceive that thing to be a threat; is it really a threat?; if it is, is it something I can handle?).

In the short time I’ve been practicing this, I’ve identified many things I had not realized I considered threats, although, of course, on the feeling level I had always known. I’m surprised by how mundane most of the threats are. Many of them are just “I feel threatened because that noise startled me.” But others are kind of embarrassing or incongruent with my self-concept. For example, I’m threatened by other people being better than me. I would find myself stiff and clearly in fight or flight when singing in a group, for instance, and I used to just nurse that anxiety for the entire practice thinking, “Fuck, I’m anxious, I can’t breathe, my singing is therefore terrible, and I must be blushing…” But with this technique, I notice the anxious symptoms and see if I can identify the “threat” that tripped them. To my shock, it was usually as simple as another person singing really well, or me not knowing how to sight read when others could. Such everyday, simple provocations! At this point, I don’t have much pride left to be embarrassed with, but it’s still humbling to see my mounta...
False assumptions and leaky abstractions in machine learning and AI safety
by David Scott Krueger (formerly: capybaralet) — 2019-06-28T04:54:47.119Z

The problems of embedded agency are due to the notion of agency implicit in reinforcement learning being a leaky abstraction.
Machine learning problem statements often makes assumptions that are known to be false, for example, assuming i.i.d. data.
Examining failure modes that result from false assumptions and leaky abstractions is important for safety, (at least) because they create additional possibilities for convergent rationality.
Attempting to enforce the assumptions implicit in machine learning problem statements is another important topic for safety research, since we do not fully understand the failure modes.
In practice, most machine learning research is done in settings where unrealistic assumptions are trivially enforced to a sufficiently high extent that it is reasonable to assume they are not violated (e.g. by the use of a fixed train/valid/test set, generated via pseudo-random uniform sampling from a fixed dataset).
We can (and probably should) do machine learning research that targets failure modes of common assumptions and methods of enforcing assumptions by (instead) creating settings in which these assumptions have the potential to be violated.
How to deal with a misleading conference talk about AI risk?
by rmoehn — 2019-06-27T21:04:32.828Z

Does it make sense to give a public response? Who would be able to do it?

The conference organizer, who had asked me to evaluate the talk, offered to interview me to set things straight. However, I don't know if that is sensible, and given my level of experience, I'm afraid I would misrepresent AI risk myself.

To be concrete: the talk was Should We Fear Intelligent Machines? by Gerald Sussman of SICP fame. He touched on important research questions and presented some interesting ideas. But much of what he said was misleading and not well-reasoned.

In response to the comments I add specifics. This is the same as I sent to the conference organizer, who had asked me for an evaluation. Note that this evaluation is separate from the interview mentioned above. The evaluation was private, the interview would be public.

Because of the low sound quality, I might have misunderstood some
statements.

Mr. Sussman touched on important research questions.

AI that can explain itself
https://arxiv.org/abs/1805.00899
https://en.wikipedia.org/wiki/Explainable_Artificial_Intelligence

Corrigibility
https://intelligence.org/research/#ET

Those who worry about AI do also worry about synthetic biology.
https://futureoflife.org/background/benefits-risks-biotechnology/
https://www.fhi.ox.ac.uk/research/research-areas/#1513088119642-44d2da6a-2ffd

Taboos – related to ‘Avoiding negative side effects’
https://blog.openai.com/concrete-ai-safety-problems/
Note that taboos rely heavily on human culture and values. Getting
those into AI is another big research area:
https://www.alignmentforum.org/posts/oH8KMnXHnw964QyS6/preface-to-the-sequence-on-value-learning
If discouraging harmful and encouraging beneficial behaviour were
easy, reinforcement learning would be the solution.

His solution approaches might be useful.

I don't know enough to judge them.

Certainly they only address a small part of the problem space,
which is laid out in:
https://arxiv.org/abs/1606.06565
https://intelligence.org/technical-agenda/
https://intelligence.org/2016/07/27/alignment-machine-learning/

He touched on some of the concerns about (strong) AI, especially the
shorter term ones.

He acknowledged AI as a threat, which is good. But he wrongly
dismissed some concerns about strong AI.

It's correct that current AI is not existential threat. But future
AI is one.

He says that it won't be an existential threa

...
Is it good practice to write questions/comments on old posts you're trying to understand? 
by Liam Donovan — 2019-06-27T09:23:01.619Z

I've recently started working through AI safety posts written on LessWrong 1-3 years ago; in doing so I occasionally have questions/comments about the material. Is it considered good practice/in line with LW norms to write these as comments on the original, old posts? One hand I can see why "necro-ing" old posts would be frowned on, but I'm not sure where else to bring it up. You can look at my comment history for examples of what I mean (before I realized it might not be a good idea)
What are principled ways for penalising complexity in practice?
by Bucky — 2019-06-27T07:28:16.850Z

Previously I asked about Solomonoff induction but essentially I asked the wrong question. Richard_Kennaway pointed me in the direction of an answer to the question which I should have asked but after investigating I still had questions.

So:

If one has 2 possible models to fit to a data set, by how much should one penalise the model which has an additional free parameter?

A couple of options which I came across were:

AIC, which has a flat facter of e penalty for each additional parameter.

BIC, which has a factor of √n penalty for each additional parameter.

where n is the number of data points.

On the one hand having a penalty which increases with n makes sense - a useful additional parameter should be able to provide more evidence the more data you have. On the other hand, having a penalty which increases with n means your prior will be different depending on the number of data points which seems wrong.

So, count me confused. Maybe there are other options which are more helpful. I don't know if the answer is too complex for a blog post but, if so, any suggestions of good text books on the subject would be great.

EDIT: johnswentworth has written a sequence which expands on the answer which he gives below.
Embedded Agency: Not Just an AI Problem
by johnswentworth — 2019-06-27T00:35:31.857Z

Requisite Background: Embedded Agency Sequence

Biology

Fast forward a few years, and imagine that we have a complete physical model of an e-coli bacteria. We know every function of every gene, kinetics of every reaction, physics of every membrane and motor. Computational models of the entire bacteria are able to accurately predict responses to every experiment we run.

Biologists say things like “the bacteria takes in information from its environment, processes that information, and makes decisions which approximately maximize fitness within its ancestral environment.” We have strong outside-view reasons to expect that the information processing in question probably approximates Bayesian reasoning (for some model of the environment), and the decision-making process approximately maximizes some expected utility function (which itself approximates fitness within the ancestral environment).

So presumably, given a complete specification of the bacteria’s physics, we ought to be able to back out its embedded world-model and utility function. How exactly do we do that, mathematically? What equations do we even need to solve?

As a computational biology professor I used to work with said, “Isn’t that, like, the entire problem of biology?”

Economics

Economists say things like “financial market prices provide the best publicly-available estimates for the probabilities of future events.” Prediction markets are an easy case, but let’s go beyond that: we have massive amounts of price data and transaction data from a wide range of financial markets - futures, stocks, options, bonds, forex... We also have some background general economic data, e.g. Fed open-market operations and IOER rate, tax code, regulatory code, and the like. How can we back out the markets’ implicit model of the economy as a whole? What equations do we need to solve to figure out, not just what markets expect, but markets’ implicit beliefs about how the world works?

Then the other half: aside from what markets expect, what do markets want? Can we map out the (approximate, local) utility functions of the component market participants, given only market data?

Neuro/Psych/FAI

Imagine we have a complete model of the human connectome. We’ve mapped every connection in one human brain, we know the dynamics of every cell type. We can simulate it all accurately enough to predict experimental outcomes.

Psychologists (among...
Apocalypse, corrupted
by Stuart_Armstrong — 2019-06-26T13:46:05.548Z

Epistemic status: don't take it seriously

In a post apocalyptic setting, the world would be run by the socially skilled and the well connected, with corruption and nepotism ruling.

I say that at the start, because I've been trying to analyse the attraction of post-apocalyptic settings: why do we like them so much? Apart from the romanticism of old ruins, four things seem to stand out:

Competence rewarded: the strong and the competent are the ones ruling, or at least making things happen. That must be the case, or else how could humans survive the new situation, where all luxuries are gone?

Clear conflict: all the heroes are in it together, against some clear menace (evil tribe or leader, zombies, or just the apocalypse itself).

Large freedom of action: instead of fitting into narrow jobs and following set bureaucratic procedures, always being careful to be polite, and so on, the heroes can let loose and do anything as long as it helps their goal.

Moral lesson: the apocalypse happened because of some failing of past humans, and everyone agrees what they did wrong. "If only we'd listened to [X]!!"

(Some of these also explain the attraction of past "golden ages".)

And I can feel the draw of all of those things! There a definite purity and attractiveness to them. Unfortunately, in a real post-apocalyptic setting, almost all of them would be false. For most of them, we're much closer to the ideal today than we would be in a post-apocalyptic world.

First of all, nepotism, corruption, and politics. The human brain is essentially designed for tribal politics, above all else. Tracking who's doing well, who's not, what coalition to join, who to repudiate, who to flatter, and so on - that's basically why our brains got so large. Tribal societies are riven with that kind of jostling and politics. We now live in an era where a lot of us have the luxury of ignoring politics at least some of the time. That luxury would be gone after an apocalypse; with no formal bureaucratic structures in place, our survival would depend on who we got along with, and who we pissed off. Competence might get rewarded - or it might get you singled out and ostracised (and ostracised = dead, in most societies). Influential groups and families would rule the roost, and most of the conflict would be internal. Forget redressing any injustice you'd be victim of; if you're not popular, you'll never ha

...
What does the word "collaborative" mean in the phrase "collaborative truthseeking"?
by Zack_M_Davis — 2019-06-26T05:26:42.295Z

I keep hearing this phrase, "collaborative truthseeking." Question: what kind of epistemic work is the word "collaborative" doing?

Like, when you (respectively I) say a thing and I (respectively you) hear it, that's going to result in some kind of state change in my (respectively your) brain. If that state change results in me (respectively you) making better predictions than I (respectively you) would have in the absence of the speech, then that's evidence for the hypothesis that at least one of us is "truthseeking."

But what's this "collaborative" thing about? How do speech-induced state changes result in better predictions if the speaker and listener are "collaborative" with each other? Are there any circumstances in which the speaker and listener being "collaborative" might result in worse predictions?
Epistemic Spot Check: The Role of Deliberate Practice in the Acquisition of Expert Performance
by Elizabeth — 2019-06-25T23:00:00.689Z

Epistemic spot checks typically consist of references from a book, selected by my interest level, checked against either the book’s source or my own research. This one is a little different that I’m focusing on a single paragraph in a single paper. Specifically as part of a larger review I read Ericsson, Krampe, and Tesch-Römer’s 1993 paper, The Role of Deliberate Practice in the Acquisition of Expert Performance (PDF), in an attempt to gain information about how long human beings can productivity do thought work over a time period.

This paper is important because if you ask people how much thought work can be done in a day, if they have an answer and a citation at all, it will be “4 hours a day” and “Cal Newport’s Deep Work“. The Ericsson paper is in turn Newport’s source. So to the extent people’s beliefs are based on anything, they’re based on this paper.

In fact I’m not even reviewing the whole paper, just this one relevant paragraph:

When individuals, especially children, start practicing in a given domain, the amount of practice is an hour or less per day (Bloom, 1985b). Similarly, laboratory studies of extended practice limit practice to about 1 hr for 3-5 days a week (e.g., Chase & Ericsson, 1982; Schneider & Shiffrin, 1977; Seibel, 1963). A number of training studies in real life have compared the efficiency of practice durations ranging from 1 -8 hr per day. These studies show essentially no benefit from durations exceeding 4 hr per day and reduced benefits from practice exceeding 2 hr (Welford, 1968; Woodworth & Schlosberg, 1954). Many studies of the acquisition of typing skill (Baddeley & Longman, 1978; Dvorak et al.. 1936) and other perceptual motor skills (Henshaw & Holman, 1930) indicate that the effective duration of deliberate practice may be closer to 1 hr per day. Pirolli and J. R. Anderson (1985) found no increased learning from doubling the number of training trials per session in their extended training study. The findings of these studies can be generalized to situations in which training is extended over long periods of time such as weeks, months, and years

Let’s go through each sentence in order. I’ve used each quote as a section header, with the citations underneath it in bold.

“When individuals, especially children, start practicing in a given domain, the amount of practice is an hour or less per day”

Generaliza...