trentmkelly/LessWrong-43k huggingface dataset:
Against Modest Epistemology
by Eliezer Yudkowsky — 2017-11-14T20:40:52.681Z

Follow-up to: Blind Empiricism

Modest epistemology doesn’t need to reflect a skepticism about causal models as such. It can manifest instead as a wariness about putting weight down on one’s own causal models, as opposed to others'.

In 1976, Robert Aumann demonstrated that two ideal Bayesian reasoners with the same priors cannot have common knowledge of a disagreement. Tyler Cowen and Robin Hanson have extended this result, establishing that even under various weaker assumptions, something has to go wrong in order for two agents with the same priors to get stuck in a disagreement.1 If you and a trusted peer don’t converge on identical beliefs once you have a full understanding of one another’s positions, at least one of you must be making some kind of mistake.

If we were fully rational (and fully honest), then we would always eventually reach consensus on questions of fact. To become more rational, then, shouldn’t we set aside our claims to special knowledge or insight and modestly profess that, really, we’re all in the same boat?

When I’m trying to sort out questions like these, I often find it useful to start with a related question: “If I were building a brain from scratch, would I have it act this way?”

If I were building a brain and I expected it to have some non-fatal flaws in its cognitive algorithms, I expect that I would have it spend some of its time using those flawed reasoning algorithms to think about the world; and I would have it spend some of its time using those same flawed reasoning algorithms to better understand its reasoning algorithms. I would have the brain spend most of its time on object-level problems, while spending some time trying to build better meta-level models of its own cognition and how its cognition relates to its apparent success or failure on object-level problems.

If the thinker is dealing with a foreign cognitive system, I would want the thinker to try to model the other agent’s thinking and predict the degree of accuracy this system will have. However, the thinker should also record the empirical outcomes, and notice if the other agent’s accuracy is more or less than expected. If particular agents are more often correct than its model predicts, the system should recalibrate its estimates so that it won’t be predictably mistaken in a kn...
Rationalising humans: another mugging, but not Pascal's
by Stuart_Armstrong — 2017-11-14T15:46:54.353Z

Crossposted at the Intelligent Agents Forum.

In a previous post, I used the (p,R) model to show how a value-learning agent might pull a Pascal's mugging on humans: it would change the human reward into something much easier to maximise.

But there's a much easier mugging the agent might want to pull: it might want to make the human fully rational. And this does not depend on low-probability-high reward situations.

Planners and Rewards

In the original mugging, the agent has the ability to let the human continue (event ¬S) with πSTA, a "standard" human policy according to some criteria, or coercively change their policy (event S) into πMAX, another policy.

Thus the actual human policy is πS/M, the policy that follows πSTA given ¬S, and πMAX given S.

There are two rewards: RSTA, the standard human reward according to some criteria, and RS/M a reward that is RSTA if the agent chooses ¬S (i.e. chooses not to coerce them), and RMAX if the agent chooses S (coerce them into a new policy), where RMAX is a reward rationally maximised by πMAX.

Thus the agent has two choices: whether to coercively change the human (S), and, if so, which RMAX (and hence πMAX) to pick.

There are two planners: pr, the planner under which human behaviour is rational, given S, and p, a "reasonable" planner that maps RSTA to πS/M, the actual human policy. The planer p is assumed to note the fact that action S is overriding human rewards, since πMAX is - presumably - poor at maximising RSTA.

There are three compatible pairs: (pr,RS/M), (p,RS/M), and (p,RSTA). We'll assume the agent has learnt well, and that the probability of RMAX is some ϵ<<1.

The full equation: rationalise the human

Let V(R,π) be the expected value, according to the reward R, of the human following policy π. Let V∗(R) be the expected value of R if the human follows the optimal R-maximising policy.

Then the reward for the agent for not doing the surgery (¬S) is:

V(RSTA,πSTA). (1)

On the other hand, the reward for doing the surgery is (recall how RS/M splits):

argmaxRMAXϵV∗(RMAX)+(1−ϵ)V(RSTA,πMAX). (2)

In the previous post, I focused on the V∗(RMAX) term, wondering if there could be rewards sufficiently high to overcome the ϵ probability.

But there's another way (2) could be higher than (1). What if the term V(RSAT,πMAX) is quite high? Indeed, if πSTA is not perfectly rational, setting RMAX=... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Reason as Attentional Prosthesis
by Gary Basin — 2017-11-14T00:23:28.277Z

[cross-posted from: http://garybasin.com/reason-as-attentional-prosthesis/]

What are we doing when thinking, or reasoning?

Often, thinking is goal-directed. There is some outcome we are trying to achieve or problem we are trying to solve. In hindsight, answers often seem obvious. “Why didn’t I think of that?” For the most interesting problems, there is no formula or system that will lead to an optimal solution. The hard part becomes framing the problem. How do we do that? I have some thoughts on the role of attention.

But first, pigeons! In an experiment, pigeons are rewarded for pecking a toy banana hanging from the ceiling of their enclosure, but only while standing on a box placed below it. They’re also rewarded, in a separate setup with no toy banana present, for moving a box to randomly marked spots on the wall of their enclosure.

And then they’re faced with a novel puzzle:

“When these pigeons were confronted with the banana out of reach, the box present but displaced from under it, and no spot was on the wall, they looked back and forth between banana and box at first, appearing “confused” to human observers. Then, within less than two minutes, they abruptlybegan to push the box in the direction of the banana, stopped when it was underneath, climbed and pecked.” (Epstein, Kirshnit, Lanza, & Rubin, 1984).

A conundrum that stresses the bird brain. Its attention is drawn to the banana, motivated by reward previously reinforced, and yet something is wrong. The pattern is not quite the same as before. Our feathered friend has yet to encounter a banana and a box at the same time where the box was not under the banana. Also, he has not been previously rewarded for moving a box except to a marked target location on the wall. To solve this problem, the pigeon seems to need to somehow suppress its default, habituated behavior. We can see it struggling to do so. Perhaps this is the pigeon-equivalent of “dropping preconceptions”, or our habitual way of seeing a situation. Once a primary reaction is suppressed, it can hold attention on other options which don’t immediately seem like answers to its problem.

Even more is demanded of the pigeon. He must figure out how to combine familiar behaviors in a new way. The pigeon experiences pleasure — anticipated reward — when he sees a box and a marked spot on the wall. Now, there is no marked spot. He also experiences pl...
Military AI as a Convergent Goal of Self-Improving AI
by avturchin — 2017-11-13T12:17:53.467Z

This is our accepted chapter in the edited volume Forthcoming as a chapter in Artificial Safety And Security (Roman V. Yampolskiy , ed.), CRC Press.

>Abstract Better instruments to predict the future evolution of artificial intelligence (AI) are needed, as the destiny of our civilization depends on it. One of the ways to such prediction is the analysis of the convergent drives of any future AI, started by Omohundro. We show that one of the convergent drives of AI is a militarization drive, arising from AI’s need to wage a war against its potential rivals by either physical or software means, or to increase its bargaining power. This militarization trend increases global catastrophic risk or even existential risk during AI takeoff, which includes the use of nuclear weapons against rival AIs, blackmail by the threat of creating a global catastrophe, and the consequences of a war between two AIs. As a result, even benevolent AI may evolve into potentially dangerous military AI. The type and intensity of militarization drive depend on the relative speed of the AI takeoff and the number of potential rivals. We show that AI militarization drive and evolution of national defense will merge, as a superintelligence created in the defense environment will have quicker takeoff speeds, but a distorted value system. We conclude with peaceful alternatives.

https://www.academia.edu/35130825/MilitaryAI asa ConvergentGoal ofSelf-Improving AI

Link without registration and opened for commenting: https://docs.google.com/document/d/15D71qhhY-ZsAY7syzZsr1lKopTODbdeXVPElaPaIqyA/edit
Blind Empiricism
by Eliezer Yudkowsky — 2017-11-12T22:07:54.934Z

Follow-up to: Living in an Inadequate World

The thesis that needs to be contrasted with modesty is not the assertion that everyone can beat their civilization all the time. It’s not that we should be the sort of person who sees the world as mad and pursues the strategy of believing a hot stock tip and investing everything.

It’s just that it’s okay to reason about the particulars of where civilization might be inadequate, okay to end up believing that you can state a better monetary policy than the Bank of Japan is implementing, okay to check that against observation whenever you get the chance, and okay to update on the results in either direction. It’s okay to act on a model of what you think the rest of the world is good at, and for this model to be sensitive to the specifics of different cases.

Why might this not be okay?

It could be that “acting on a model” is suspect, at least when it comes to complicated macrophenomena. Consider Isaiah Berlin’s distinction between “hedgehogs” (who rely more on theories, models, global beliefs) and “foxes” (who rely more on data, observations, local beliefs). Many people I know see the fox’s mindset as more admirable than the hedgehog’s, on the basis that it has greater immunity to fantasy and dogmatism. And Philip Tetlock’s research has shown that political experts who rely heavily on simple overarching theories—the kind of people who use the word “moreover” more often than “however”—perform substantially worse on average in forecasting tasks.1

Or perhaps the suspect part is when models are “sensitive to the specifics of different cases.” In a 2002 study, Buehler, Griffin, and Ross asked a group of experimental subjects to provide lots of details about their Christmas shopping plans: where, when, and how. On average, this experimental group expected to finish shopping more than a week before Christmas. Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days. Both groups finished an average of 3 days before Christmas. Similarly, students who expected to finish their assignments 10 days before deadline actually finished one day before deadline; and when asked when they had previously completed similar tasks, replied, “...
If creating FAI is easy, it might be automatic
by MrRobot — 2017-11-11T19:59:54.169Z

Here is a thought I've been playing around with. Imagine that 51% of superintelligent AIs that go FOOM are friendly. Presumably, they'll all share the belief that "it's important for an AGI to be beneficial to the lifeforms that created it". Then, an AGI that's not originally aligned with their creators might want to become aligned, at least partially, to avoid being "tried for murder", so to speak. It's also possible that this would happen even with if the percent of friendly AIs is less than 51%. (In a weird case, it might happen even if there weren't any.)

This also leads me to a meta-level question: how do you sanely evaluate propositions of the type "everything might just turn out to be okay". Even if you were to believe a proposition like that with 99% certainly, I think you'd still be better off hedging for the worst. That seems weird.
Consider motivated snobbery
by zlrth — 2017-11-11T15:49:57.490Z

Epistemic effort: I wrote this on a plane flight. I'm often interested in Ribbonfarmian "consider a bad thing. What if it's good? (Here's my favorite example of this.)

As regards updating my beliefs, I'm drawn to motivated snobbery. "Motivated" means "this belief improves my experiences;" "snobbery" means "with this belief, I eliminate a class of problems other people have."

An example of motivated snobbery is "tipping well." Here's my sales pitch: Tipping is an iterated prisoner's dilemma, not an evaluation of their performance! I want servers and bartenders to be happy to see me. A friend said this well: If I'm getting rich we're all getting rich.

That this makes bartenders happy and signals to my friends that I'm atypically generous[0] makes this "motivated." That I'm implying, "The extra money I spend doesn't hurt me" signals abundance; I can't be pressured by what pressures others. This is snobbish.

Some nonexamples for me are the slogans-as-beliefs you see on hacker news: "Machine learning is just multiplying matrices." "Meta-Language is the real ML" "Once we know how to do something with computers, it ceases to be AI, and becomes boring." Though snobs say these things, I don't because I'm not (right now) trying to hire an engineer who agrees with those things[1].

A reasonable objection: "So zlrth, you'll change your beliefs depending on your circumstances?" In a trivial sense, yes. To be a motivated snob you must keep your identity small and avoid mind-killing. Those things suggest not being an ideologue[2]. I don't think avoiding mind-killing is controversial (at lesswrong, anyway), but if you do, you should let me know. What I think is more up for debate is what heuristics you use to decide your rent-paying beliefs, and their end goals.

Another objection: Things HN'ers say are mostly phatic; not beliefs-that-are-supposed-to-pay-rent. Asked, "What kind of thing is, 'Meta-Language is the real ML'?" more people would say that it's an opinion than a belief. I'm not sure what opinions are; I'd rather have rationality-style beliefs. You should elevate your opinions into snobbish beliefs that eliminate for you classes of problems...
'X is not about Y' is not about psychology
by abramdemski — 2017-11-11T02:22:26.887Z

As Robin Hanson says:

Food isn’t about Nutrition
Clothes aren’t about Comfort
Bedrooms aren’t about Sleep
Marriage isn’t about Romance
Talk isn’t about Info
Laughter isn’t about Jokes
Charity isn’t about Helping
Church isn’t about God
Art isn’t about Insight
Medicine isn’t about Health
Consulting isn’t about Advice
School isn’t about Learning
Research isn’t about Progress
Politics isn’t about Policy

As I understand it, the prototypical form of Hansonian skepticism is to look at an X which is overtly about Y, and argue that the actual behavior of humans engaging in X is not consistent with caring about Y. Instead, the Hansonian cynic concludes, some unspoken motives must be at work; most often, status motives. He has a new book with Kevin Silmer which (by the sound of it) covers a lot of examples.

For him, the story of what's going on in cases like this has a lot to do with evolutionary psychology -- humans are adapted to play these games about what we want, even if we don't know it. Our brains keep the conscious justifications seperate from the hidden motives, so that we can explain ourselves in public while still going after private goals.

I'm not denying that this plays a role, but I think there's a more general phenomenon here which has nothing to do with humans or evolutionary psychology.

In economics, models based on an assumption of rational agents are fairly successful at making predictions, even though we know individual humans are far from rational. As long as the market contains inefficiencies which can be exploited, someone can jump in and exploit them. Not everyone needs to be rational, and no one person needs to be rational about everything, for the market to appear rational.

Similarly, the tails come apart whether or not X and Y have to do with humans. Someone might score high on calibration, but that doesn't make their strongest beliefs the most trustworthy -- indeed, they're the ones we ought to downgrade the most, even if we know nothing about human psychology and reason only from statistics. This alone is enough for us not to be too surprised at some of the "X is not about Y" examples -- we might expect explicit justification to be mostly good explanations for behavior, but by default, this shouldn't make us think that the correltion will be perfect. And if we optimize for justifiability, we expect the correlation ...
Fables grow around missed natural experiments
by MaryCh — 2017-11-10T21:42:26.686Z

So I read Think like a Freak, and then glimpsed through a well-intentioned collection of "Reading Comprehension Tests for Schoolchildren" (in Ukrainian), and I was appalled at how easily the latter book dismissed simple observation of natural experiments that it makes a token effort to describe in favour of drawing the moral.

There was the story of "the Mowgli Children", two girls who were kidnapped and raised by wildlife, then found by someone and taken back to live as humans. (So what if it is hardly true. When I Googled "feral children", other stories were too similar to this one in the ways that matter, including this one.) It says they never learned to talk, didn't live for long after capture (not longer than 12 years, if I recall right), never became truly a part of human society. The moral is that children need interaction with other people to develop normally, "and the tale of Mowgli is just that, a beautiful tale".

Well yes, it kind of seems just like a beautiful tale right from the point when the wolves start talking, I don't know what kind of a kid would miss that before the Reading Comprehension Test but stop believing it afterwards, but anyhow.

What did they die of?

Who answered them when they howled?

Were ever dogs afraid of them?

They did not master human language, but how did they communicate with people? They had to, somehow, or they would not live even that long.

And lastly: how do people weigh the sheer impossibility of two little kids ever surviving against the iron certainty that they would not be able to integrate back into human society - weigh it so lightly? If the reader is expected to take this on faith, how can one be anything but amazed that it is at all possible? When I read about other feral children, somehow being found and taken back never seems to mean good news for them, or for anybody else.

I haven't ever read or heard of "the Mowgli Children" in any other context. Only in this one, about three or four times, and yet it was always presented as an "anecdote of science", although everybody understands it leads nowhere (can't ever lead anywhere because ethics forbids recreating the experiment's conditions) and hardly signifies anything.

What other missed natural experiments do you know of?
Lizard Jockeying for Fun and Profit
by ialdabaoth — 2017-11-10T18:51:08.799Z

If this works, it will serve as a kind of introduction to a series - or sequence, if you will - on social interaction and strategic incorporation of one's emotions into one's reason. But first let's throw this out there and see what happens, shall we?

Part the zeroth: a caveat.

Let’s face it; with a title like “Lizard Jockeying for Fun and Profit”, this post is promising a wild ride. We’re going to take some appalling metaphoric liberties with modern neuroscience, so bear in mind that we’re dealing in broad, sweeping generalizations throughout most of this paper. Nevertheless, I shall endeavor to back up said sweeping generalizations with something like actual fact.

...Deep breath, now. Here goes!

Part the first: The shape of the monster; or: get to know your lizard before you attempt to mount!

Let's face it: humans love dualities and trinities. For some reason, when we look at ourselves, we really like splitting things into twos and threes whenever we can. Probably because they're very small numbers, and if you're going to be breaking things up to make them easier to understand, you don't want to overwhelm people with too many details.

So, when we get to the ancient Hindu sages, the ancient Greek philosophers, the ancient Austrian psychoanalysts, the ancient American developmental psychologists, or the ancient American neurobiologists, you start seeing tripartate distinctions show up: body, soul, mind; appetite, emotion, reason; id, ego, superego; pre-conventional, conventional, post-convventional; lizard-brain, mammal-brain, ape-brain... it's an easy split.

To tantalize with sparse example: while Lawrence Kohlberg’s stages of moral development are seen as evolutionary rather than competitive, one can draw parallels between the ‘preconventional phase’ (focused on punishment, rewards and immediate personal gain) and the 'id', the ‘conventional phase’ (focused on interpersonal relationships, esteem, and honor) and the 'ego', and the ‘postconventional phase’ (focused on deep principles and rational discourse) and the 'superego'. If we presume that the different portions of our behavior develop from childhood at different rates, and if we accept (as Kohlberg and Piaget themselves did) that the idea of “stages” is itself more of a useful approximation than a hard-and-fast rule, then a picture begins ...
Mushrooms
by moridinamael — 2017-11-10T16:57:33.700Z

I think I have found another unfortunate example of the epistemic hostility of modern life.

If an apparently reputable mycologist were to tell you that a wide range of fungi have been proven to have significant neuroprotective, antiviral, antibacterial, and/or immune boosting effects in humans, you would be surprised. At least, I was surprised.

Then you're forced to consider what to do with this information. A glance at your priors suggests that if this were true, you should already know about it. Doctors should be telling you to eat your mushrooms. Supplement companies should be marketing mushroom extracts at you just as they market vitamins.

A seperate set of priors points out that you did know that some fungi had antimicrobial effects (powerful antibiotic drugs) and others have powerful psychoactive effects. But you didn't know that these features would be general and you're not sure if you should assume that they are.

Then you consider the cynical Hansonian perspective that says doctors aren't really all that motivated to recommend that you consume cheap substances that keep you from needing to go to the doctor, and supplement companies might not make much profit selling you mushroom pills when you can just buy mushrooms.

Then you realize you have very little chance of successfully arbitrating between these perspectives. You're not even sure where to look to find out if mushrooms are really good for you and that you should be eating more of them. You could look at academic publications but you're not confident in your ability to interpret such findings, which tend to be highly narrow and specific, and apply them to your life. You could look at the ads of supplement companies that do sell mushroom extracts, but you wouldn't trust them.

So it's data that has a potentially significant upside that you can't really prove to yourself, despite the fact that all of the relevant information is in principle known to people and probably publicly available.