trentmkelly/LessWrong-43k huggingface dataset:
An Increasingly Manipulative Newsfeed
by Michaël Trazzi — 2019-07-01T15:26:42.566Z

Co-written with Stuart Armstrong

Treacherous turn vs sordid stumble

Nick Bostrom came up with the idea of a treacherous turn for smart AIs.

while weak, an AI behaves cooperatively. When the AI is strong enough to be unstoppable it pursues its own values.

Ben Goertzel criticised this thesis, pointing out that:

for a resource-constrained system, learning to actually possess human values is going to be much easier than learning to fake them. This is related to the everyday observation that maintaining a web of lies rapidly gets very complicated.

This argument has been formalised into the sordid stumble:

An AI that lacks human desirable values will behave in a way that reveals its human-undesirable values to humans before it gains the capability to deceive humans into believing that it has human-desirable values.

The AI is too dumb to lie (well)

The sordid stumble describes a plausible sounding scenario for how an AI develops capabilities. Initially, the AI doesn't know our values, and doesn't know us. Then it will start to learn our values (and we'll be checking up on how well it does that). It also starts to learn about us.

And then, once it's learnt some about us, it may decide to lie - about its values, and/or about its capabilities. But, like any beginner, it isn't very good at this initially: its lies and attempts at dissembling are laughably transparent, and we catch it quickly.

In this view, the "effective lying" is a tiny part of policy space, similar to the wireheading in this example. To hit it, the AI has to be very capable; to hit it the first time it tries without giving the game away, the AI has to be extraordinarily.

So, most likely, either the AI doesn't try to lie at all, or it does so and we catch it and sound the alarm[1].

Lying and concealing... from the very beginning

It's key to note that "lying" isn't a fundamentally defined category, and nor is truth. What is needed is that the AI's answer promotes correct understanding in those interacting with it. And that's a very different kettle of fish being shot in that barrel.

This opens the possibility that the AI could be manipulating us from the very beginning, and would constantly learn to do so better.

The (manipulative) unbiased newsfeed

Imagine that there was some company that could somehow choose the newsfeed of billions of people across the world (I know, completely science-fictionny, but bear with me... ,
What's the most "stuck" you've been with an argument, that eventually got resolved?
by Raemon — 2019-07-01T05:13:26.743Z

I was trying to write up a post about getting stuck in arguments, where you thought something was obvious, and said something like "A, therefore, B." And your partner says "but, X, therefore Y!"

And from you perspective, A is obviously true, and B obviously follows from A. And X and Y just don't seem relevant to you. But somehow, to your conversation partner, A and B either aren't obvious at all, or just seem... irrelevant somehow. And X and Y seem like the important things.

I feel like this happens often to me, but I didn't carefully keep track of them at the time. I'll try checking in with people I've disagreed with in the past to see if they can help jog my memory. But I'm curious if anyone has ran into this phenomenon?

A somewhat extreme but concrete example might be a consequentialist and a deontologist arguing about ethics, where the Connie the consequentialist says "We should throw the switch in the trolley problem to save lives" and Denny the Deontologist says "But you're murdering the guy on the tracks!" and both of them are coming at the problem from such different perspectives that they can't make any headway.

I'd ideally like examples that each involve positions that are fairly common on LessWrong.

(The important bit isn't precisely the "A therefore B", "X therefore Y" pattern, so much as the general phenomenon of feeling like the two participants were somehow managing to completely miss each other. Ideally, examples where the people eventually ended up on the same page, hopefully identifying the moment when the argument became "unstuck.")
The Competence Myth
by Davis_Kingsley — 2019-06-30T18:55:49.014Z

When I was a kid, everything seemed sort of goofy, but there was also a sense that it wasn't really supposed to be competent. Yes, a lot of stuff targeted at kids was obviously dumb and bad, but it was for kids after all -- grown-ups knew what they were doing, and once we were grown up we wouldn't have to deal with that!

Once I became a teenager, it became pretty obvious that grown-ups didn't always know what they were doing either. [1] But there was still a sense that at the next level people would finally know what was going on, and that sense persisted for some time - perhaps soon it was "college students", then it was "professors", then it was "people in the private sector", then it was "experienced workers at more professional organizations"... at every level, after every hurdle, it turned out that things were about as slapdash and incompetent as before. It wasn't always the same exact patterns, but there was always something going wrong, and there was never the sense of "okay, now I can fully Trust the System".

Maybe the final straw came when I read or heard something from (IIRC) a Navy SEAL, who said that even in the SEAL teams there were people who were incompetent - in other words, even after extremely stringent training and selection processes that would be illegal for any normal organization to implement, they were still unable to achieve that fabled "okay, finally we're at a stage where everyone knows what they're doing".

Now, I'm not saying every single group is always and forever going to be incompetent - but what I am saying is that I wasted a lot of time thinking "oh, things are messed up now but once I reach this next educational/professional milestone, everything will be fine!" In point of fact that's just not the case, and you should plan accordingly. Almost regardless of what level of vetting and selection you implement, there will be some people who slip through the cracks, and there will be some systems or processes that aren't so good. [2] Lastly, if anyone can think of any large groups or organizations where this isn't the case, please tell me what they are! I'd love to be proven wrong on this.

[1] A school administrator once accused me of drawing something he described as "a pagan symbol from The Da Vinci Cod...
Is AlphaZero any good without the tree search?
by Steven Byrnes — 2019-06-30T16:41:05.841Z

One component of AlphaZero is a neural net which takes a board position as input, and outputs a guess about how good the position is and what a good next move would be. It combines this neural net with Monte Carlo Tree Search (MCTS) that plays out different ways the game could go, before choosing the move. The MCTS is used both during self-play to train the neural net, and during competitive test-time. I'm mainly curious about whether the latter is necessary.

So my question is: Once you have the fully-trained AlphaZero system, if you then turn off the MCTS and just choose moves directly with the neural net policy head, is it any good? Is it professional-level, amateur-level, child-level?

(I think this would be a fun little data-point related to discussions of how powerful an AI can be with and without mesa-optimization / search-processes using a generative environmental model.)
Being Wrong Doesn't Mean You're Stupid and Bad (Probably)
by Zack_M_Davis — 2019-06-29T23:58:09.105Z

Sometimes, people are reluctant to admit that they were wrong about something, because they're afraid that "You are wrong about this" carries inextricable connotations of "You are stupid and bad." But this behavior is, itself, wrong, for at least two reasons.

First, because it's evidential decision theory. The so-called "rationalist" "community" has a lot of cached clichés about this! A blank map does not correspond to a blank territory. What's true is already so; owning up to it doesn't make it worse. Refusing to go to the doctor (thereby avoiding encountering evidence that you're sick) doesn't keep you healthy.

If being wrong means that you're stupid and bad, then preventing yourself from knowing that you were wrong doesn't stop you from being stupid and bad in reality. It just prevents you from knowing that you're stupid and bad—which is an important fact to know (if it's true), because if you don't know that you're stupid and bad, then it probably won't occur to you to even look for possible interventions to make yourself less stupid and less bad.

Second, while "You are wrong about this" is evidence for the "You are stupid and bad" hypothesis if stupid and bad people are more likely to be wrong, I claim that it's very weak evidence. (Although it's possible that I'm wrong about this—and if I'm wrong, it's furthermore possible that the reason I'm wrong is because I'm stupid and bad.)

Exactly how weak evidence is it? It's hard to guess directly, but fortunately, we can use probability theory to reduce the claim into more "atomic" conditional and prior probabilities that might be easier to estimate!

Let W represent the proposition "You are wrong about something", S represent the proposition "You are stupid", and B represent the proposition "You are bad."

By Bayes's theorem, the probability that you are stupid and bad given that you're wrong about something is given by—

P(S,B|W)=P(W|S,B)P(S,B)P(W|S,B)P(S,B)+P(W|S,¬B)P(S,¬B)+P(W|¬S,B)P(¬S,B)+P(W|¬S,¬B)P(¬S,¬B)

For the purposes of this calculation, let's assume that badness and stupidity are statistically independent. I doubt this is true in the real world, but because I'm stupid and bad (at math), I want that simplifying assumption to make the algebra easier for me. That lets us unpack the conjunctions, giving us—

P(S,B|W)=P(W|S,B)P(S)P(B)P(W|S,B)P(S)P(B)+P(W|S,¬B)P(S)P(¬B)+P(W|¬S,B)P(¬S)P(B)+P(W|¬S,¬B)P(¬S)P(¬B)

This e

...
Do children lose 'childlike curiosity?' Why?
by Raemon — 2019-06-29T22:42:36.856Z

A common story goes:

Young children love asking 'why', and they often have an earnest curiosity about it that is rare in adults. Something about the process of growing up seems to cause that childlike curiosity to stagnate.

There's a lot of compelling explanations about societal norms that actively stamp out that curiosity (i.e. school training kids to conform and regurgitate facts, parents subtly punishing kids for asking questions, etc). It seems likely to me that these are at least part of the story.

But it also wasn't obvious that they were the whole story. I could easily imagine it also just being the case that small children are optimized for learning and older humans are optimized for doing and the brain automatically shifts away from it. [edit: or that this whole thing is imagined]

Are there any cross-cultural studies that do anything to check how this phenomenon varies, depending on upbringing?
How can we measure creativity?
by Elizabeth — 2019-06-29T22:30:47.805Z

Status: have spent about two hours on this.

As part of measuring how marginal productivity changes over time, I need to know how to assess creativity. One promising test for that is the Torrance Tests of Creative Thinking, in which subjects are given an open ended prompt, and graded on their answers. Answers are evaluated by a human being for fluency, originality, abstractness of titles, elaboration and resistance to premature closure (sample questions for the curious). But does the TTCT predict anything we actually care about?

The creator of the tests studied their predictive ability in a longitudinal study that lasted 50 years so far. The 40 year follow up showed good-for-social-sciences correlation between childhood TTCT scores and adult creative achievement, although IQ had a stronger correlation. The 50 year follow up (conducted by different experimenters) found no correlation between score and "public achievement" unless combined with IQ. Given that these studies were subject to the usual social science weaknesses, multiplied by 50 years and subjective grading, I do not count this as strong evidence.

A smaller study in Brazil found adulthood scores of recognized creative achievers and non-achievers to vary.

Basic googling found more academics saying both disparaging (beginning of page 310) and encouraging things.

My default assumption is that psychometric tests are invalid, and this evidence isn't enough to make me change my mind. But I don't have anything better to use for my actual goal, which is a measurable task that taxes creativity and *nothing else*, and this has a certain face validity to it. Does anyone have information to sway on the validity of the test, or an alternative test to use?
An unconfortable epistemic position regarding "Verb Magic".
by Hijol — 2019-06-29T03:38:30.085Z

Hello Guys,

This post will act as a bottle thrown in the sea, a "Is anyone like me anywhere ?" as well as an occasion, i hope, to discuss a more serious topic with you. Some claims made here might push on your "pseudo-science" button, please read until the end.

Please allow me to start with a brief (not so brief) testimonial in order for you to grab the topic.

I'm a 21 baguette french person.

I consider myself as a rationalist, i believe in demonstration by facts and I believe I don't have any superstition like telepathy, parapsychology or flat-earth BS. I love all kind of science and deductive/inductive thinking in general. I'm basically passioned by anything that can be interesting.

I am a student in psychology, i'm aware of the scientific methodology and aware of the ton of cognitive bias the human mind can follow. I like physics, astrophysics, human sciences as well as "hard" sciences.

But, and some of you might stop your reading here for what i'm about to say, I recently dove into occultism. please wait and let me explain.

First of all, it's historically speaking a cool subject to consider, i read a lot of books about alchemy, tarot, mistery cults etc... There's a lot to learn about epistemology, theology and more. But it developped something else in me. Some kind of beliefs in some king of... magic.

Again, i'm not in a new age type of thinking like "your intuition is the path everything else is wrong", I'm an atheist and the magic i'm talking about is more like something I got from a lot of philosophical thinking.

To clarify, I believe in "The Verb" kind of magic, the symbol and the ritual. It can look something like a thing A. Jodorowsky calls "psychomagic" (what a marketing term again). I started to believe that most of the things we called magic in the history of occultism is just a beautiful and poetic way to talk about advanced symbolic speech and act, which can have an enormous impact on the psychologic, emotionnal, and spiritual mindset of the person who performs it.

We can find this in franc-maçonnerie, the ritual has no supernatural nature but the power which emanates from the performance engraves a lot of things in your person, and this kinda looks like what we called sorcery at some point in history. In the same mindset, i started sudying tar...
Conceptual Problems with UDT and Policy Selection
by abramdemski — 2019-06-28T23:50:22.807Z

Abstract

UDT doesn't give us conceptual tools for dealing with multiagent coordination problems. There may have initially been some hope, because a UDT player can select a policy which incentivises others to cooperate, or because UDT can reason (EDT-style) that other UDTs are more likely to cooperate if it cooperates itself, or other lines of thought. However, it now appears that this doesn't pan out, at least not without other conceptual breakthroughs (which I suggest won't look that much like UDT). I suggest this is connected with UDT's difficulties handling logical uncertainty.

Introduction

I tend to mostly think of UDT as the ideal, with other decision theories being of interest primarily because we don't yet know how to generalize UDT beyond the simplistic domain where it definitely makes sense. This perspective has been increasingly problematic for me, however, and I now feel I can say some relatively concrete things about UDT being wrong in spirit rather than only in detail.

Relatedly, in late 2017 I made a post titled Policy Selection Solves Most Problems. Policy selection is the compromise solution which does basically what UDT is supposed to do, without the same degree of conceptual elegance, and without providing the hoped-for clarity which was a major motivation for studying these foundational problems. The current post can be seen as a follow-up to that, giving an idea of the sort of thing which policy selection doesn't seem to solve.

The argument can also be thought of as an argument against veil-of-ignorance morality of a certain kind.

I don't think any of this will be really surprising to people who have been thinking about this for a while, but, my thoughts seem to have recently gained clarity and definiteness.

Terminology Notes/References

UDT 1.0, on seeing observation obs, takes the action ai which maximizes the expected utility of "my code outputs action ai on seeing observation obs", with expected value evaluated according to the prior.

UDT 1.1, on seeing observation obs, takes the action ai which the globally optimal policy (according to the prior) maps obs to. This produces the same result as UDT 1.0 in many cases, but ensures that the agent can hunt stag with itself.

UDT 2 is like UDT 1.1, except it (1) represents policies as programs rather than input-output mappings, and (2) dynamically decides ho...
Whence decision exhaustion?
by Gordon Seidoh Worley — 2019-06-28T20:41:47.987Z

Many people experience something we might call decision or executive exhaustion: after making a lot of decisions, it can be hard to make more decisions and to exert "willpower". Yet, this seems odd because we are constantly making decisions all the time in some sense, choosing to do what we do over everything else we could have otherwise done. So, what and why do we sometimes get exhausted of making decisions when most of the time we do not?

Some notes to consider in answering:

Some people seem to experience this from all decisions and are worn out after dozens of minutes of being awake.
Some people seem to never experience this.
Exhausting decisions seem more salient or like they require more deliberate thought than ones that are not exhausting. Non-exhausting decisions feel automatic.
Food and rest (but not a full nights sleep) helps some people recover decision function but not everyone seems to respond to this over short enough timescales for it to be useful for recovering functionality within the day.
Aligning a toy model of optimization
by paulfchristiano — 2019-06-28T20:23:51.337Z

Suppose I have a magic box Opt that takes as input a program U:{0,1}n→R, and produces Opt(U)=argmaxxU(x), with only n times the cost of a single evaluation of U. Could we use this box to build an aligned AI, or would broad access to such a box result in doom?

This capability is vaguely similar to modern ML, especially if we use Opt to search over programs. But I think we can learn something from studying simpler models.

An unaligned benchmark

(Related.)

I can use Opt to define a simple unaligned AI (details omitted):

Collect data from a whole bunch of sensors, including a "reward channel."
Use Opt to find a program M that makes good predictions about that data.
Use Opt to find a policy π that achieves a high reward when interacting with M.

This isn't a great design, but it works as a benchmark. Can we build an aligned AI that is equally competent?

(I haven't described how Opt works for stochastic programs. The most natural definition is a bit complicated, but the details don't seem to matter much. You can just imagine that it returns a random x that is within one standard deviation of the optimal expected value.)

Competing with the benchmark

(Related.)

If I run this system with a long time horizon and a hard-to-influence reward channel, then it may competently acquire influence in order to achieve a high reward.

We'd like to use Opt to build an AI that acquires influence just as effectively, but will use that influence to give us security and resources to reflect and grow wiser, and remain responsive to our instructions.

We'd like the aligned AI to be almost as efficient. Ideally the proportional overhead would converge to 0 as we consider more complex models. At worst the overhead should be a constant factor.

Possible approach

(Related.)

My hope is to use Opt to learn a policy π+ which can answer questions in a way that reflects "everything π knows." This requires:

Setting up an objective that incentivizes π+ to give good answers to questions.
Arguing that there exists a suitable policy π+ that is only slightly more complicated than π.

If we have such a π+, then we can use it to directly answer questions like "What's the best thing to do in this situation?" The hope is:

Its answers can leverage everything π knows, and in particular all of π's knowledge about how to acquire influence. So using π+ in this way

...
Book Review: Why Are The Prices So Damn High?
by sarahconstantin — 2019-06-28T19:40:00.643Z

Economist Alex Tabarrok has recently come out with a short book, “Why are the prices so Damn High”, available in full PDF here.

Since the 1950’s, the inflation-adjusted cost of physical goods has fallen since the 1950’s, and the cost of food has stayed about the same. But the cost of education, professional services, and healthcare has risen dramatically, despite those sectors not producing much improvement. Why?

The traditional economic explanation for the rising cost of services is the Baumol Effect. Some sectors, like manufacturing, are subject to efficiency improvements over time as technology improves; the more we automate the production of goods, the cheaper they get. Other sectors are intrinsically harder to automate, so they don’t get cheaper over time. For instance, it takes the same number of musicians the same number of time to play a symphony as it did in 1950. So, as a proportion of the average person’s paycheck, the cost of intrinsically un-automatable things like live concerts must rise relative to the cost of automatable things like manufactured goods.

Tabarrok doesn’t cover housing in his book, but home prices have also been rising since the 1970’s and I’ve seen the Baumol effect deployed to explain rising housing costs as well. “Land is the one thing they’re not making any more of” — for the most part, technological improvements don’t increase the quantity of livable land, so if technology makes some sectors more efficient and drives costs down, land will become relatively more expensive.

My Beef With Baumol

My preconception coming into the book was that the Baumol effect doesn’t actually answer the question. Why are healthcare, professional services, and education intrinsically hard to make more efficient? It’s prima facie absurd to say that medicine is just one of those things that technology can’t improve — the biomedical industry is one of the biggest R&D endeavors in the whole economy! So why is it obvious that none of that innovation can make medicine cheaper? If it’s not making medicine cheaper, that’s an empirical fact that deserves explanation, and “it’s the Baumol effect” doesn’t actually answer the “why” question.

The same holds true for the other industries, even housing to some degree. While it’s true that the amount of land on Earth is fixed (modulo landfill) and the amount of space in Manhattan is fixed, there’s also the ...