trentmkelly/LessWrong-43k huggingface dataset:
What are human values? - Thoughts and challenges
by Chris_Leong — 2019-11-02T10:52:51.585Z

Epistemic Status: Rough outline of ideas

The first step in achieving a goal is knowing what you want. But in order to answer this question, we first need to know what it means to want. Do we care more about your desires at the point of the decision or at the point of the experience? Do we care more about what you want emotionally or what you want cognitively? Ultimately, there's no real meaning to the word "want" inscribed in the universe, there's just a whole bunch of related meanings clustered under the same term. I'll try to break this down, but unfortunately I can only sketch a rough non-empirically based model as this post really just needs to be written by a neuroscientist or psychologist.

Our brain seems to consist of three main subsystems. Firstly, we have emotional systems like pleasure, pain, desire, disinterest, rightness and wrongness. These trigger at various times: contemplation of a possible choice or outcome, after locking in a decision, after learning about the outcome, whilst experiencing the outcome and in self-reflection afterwards. We may switch from positive to negative and back again at different stages and this may be because our preferences have changed or it may just be a consistent conflict between our preferences. For example, we may feel inspired when considering a physical challenge, hate every moment of the experience and then feel a real sense of achievement when we are done.

Secondly, we have the cognitive components. These can be explicit goals ("I must achieve X"), things to be avoided ("I must not fail"), moral imperatives ("I have an obligation to protect X") and decisions ("I will prioritise my long term happiness over temporary pleasures"). Again, these evaluations may change during an experience or afterwards and it's not always clear whether this is us updating our preferences, keeping our preference and learning more information or some of our preferences being fake and merely for social signalling.

Thirdly, we have intuition or a sense that certain actions will be good or bad. This may oppose our explicit cognitive components, for example, a man who is consciously trying to become a lawyer, but subconsciously sabotaging themselves because they think they'd hate it. This is separate from the emotional system as we can have an intuition that we'll hate something without feeling dread about the possibility of it occurring...
AlphaStar: Impressive for RL progress, not for AGI progress
by orthonormal — 2019-11-02T01:50:27.208Z

DeepMind released their AlphaStar paper a few days ago, having reached Grandmaster level at the partial-information real-time strategy game StarCraft II over the summer.

This is very impressive, and yet less impressive than it sounds. I used to watch a lot of StarCraft II (I stopped interacting with Blizzard recently because of how they rolled over for China), and over the summer there were many breakdowns of AlphaStar games once players figured out how to identify the accounts.

The impressive part is getting reinforcement learning to work at all in such a vast state space- that took breakthroughs beyond what was necessary to solve Go and beat Atari games. AlphaStar had to have a rich enough set of potential concepts (in the sense that e.g. a convolutional net ends up having concepts of different textures) that it could learn a concept like "construct building P" or "attack unit Q" or "stay out of the range of unit R" rather than just "select spot S and enter key T". This is new and worth celebrating.

The overhyped part is that AlphaStar doesn't really do the "strategy" part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don't look creative or even especially reactive to opponent strategies.

That's because there's no representation of causal thinking - "if I did X then they could do Y, so I'd better do X' instead". Instead there are many agents evolving together, and if there's an agent evolving to try Y then the agents doing X will be replaced with agents that do X'. But to explore as much as humans do of the game tree of viable strategies, this approach could take an amount of computing resources that not even today's DeepMind could afford.

(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents' units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn't get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it d...
Chris Olah’s views on AGI safety
by evhub — 2019-11-01T20:13:35.210Z

Note: I am not Chris Olah. This post was the result of lots of back-and-forth with Chris, but everything here is my interpretation of what Chris believes, not necessarily what he actually believes. Chris also wanted me to emphasize that his thinking is informed by all of his colleagues on the OpenAI Clarity team and at other organizations.

In thinking about AGI safety—and really any complex topic on which many smart people disagree—I’ve often found it very useful to build a collection of different viewpoints from people that I respect that I feel like I understand well enough to be able to think from their perspective. For example, I will often try to compare what an idea feels like when I put on my Paul Christiano hat to what it feels like when I put on my Scott Garrabrant hat. Recently, I feel like I’ve gained a new hat that I’ve found extremely valuable that I also don’t think many other people in this community have, which is my Chris Olah hat. The goal of this post is to try to give that hat to more people.

If you’re not familiar with him, Chris Olah leads the Clarity team at OpenAI and formerly used to work at Google Brain. Chris has been a part of many of the most exciting ML interpretability results in the last five years, including Activation Atlases, Building Blocks of Interpretability, Feature Visualization, and DeepDream. Chris was also a coauthor of “Concrete Problems in AI Safety.”

He also thinks a lot about technical AGI safety and has a lot of thoughts on how ML interpretability work can play into that—thoughts which, unfortunately, haven’t really been recorded previously. So: here’s my take on Chris’s AGI safety worldview.

The benefits of transparency and interpretability

Since Chris primarily works on ML transparency and interpretability, the obvious first question to ask is how he imagines that sort of research aiding with AGI safety. When I was talking with him, Chris listed four distinct ways in which he thought transparency and interpretability could help, which I’ll go over in his order of importance.

Catching problems with auditing

First, Chris says, interpretability gives you a mulligan. Before you deploy your AI, you can throw all of your interpretability tools at it to check and see what it actually learned and make sure it learned the right thing. If it didn’t—if you find that it’s learned some sort of potentially dangerous proxy, fo...
Explaining Visual Thinking
by Teach — 2019-11-01T20:00:54.037Z

One of my students asked me how visual thinking works. As someone who does not think in pictures, I was a bit stumped on how to explain it because I can't simulate visual thinking in my mind (I usually try to conceptualise and simulate things I haven't experienced in order to understand them better). I'm actually really interested now, myself, in understanding the mechanisms of visual thinking, and how it all works.

Are any of you here visual thinkers? How would you explain the way your visual thinking occurs in your mind to someone so they could conceptualise it in terms of both the processes and the sensations?
The Curse Of The Counterfactual
by pjeby — 2019-11-01T18:34:41.186Z

The Introduction

The Curse of the Counterfactual is a side-effect of the way our brains process is-ought distinctions. It causes our brains to compare our past, present, and future to various counterfactual imaginings, and then blame and punish ourselves for the difference between reality, and whatever we just made up to replace it.

Seen from the outside, this process manifests itself as stress, anxiety, procrastination, perfectionism, creative blocks, loss of motivation, inability to let go of the past, constant starting and stopping on one goal or frequent switching between goals, low self-esteem and many other things. From the inside, however, these counterfactuals can feel more real to us than reality itself, which can make it difficult to even notice it's happening, let alone being able to stop it.

Unfortunately, even though each specific instance of the curse can be defused using relatively simple techniques, we can’t just remove the parts of our brain that generate new instances of the problem. Which means that you can’t sidestep the Curse by imagining yet another counterfactual world: one in which you believe you ought to be able to avoid falling into its trap, just by being smarter or more virtuous!

Using examples derived from my client work, this article will show how the Curse operates, and the bare bones of some approaches to rectifying it, with links to further learning materials. (Case descriptions are anonymized and/or composites; i.e., the names are not real, and identifying details have been changed.)

The Disclaimer

To avoid confusion between object-level advice, and the meta-level issue of “how our moral judgment frames interfere with rational thinking”, I have intentionally omitted any description of how the fictionalized or composite clients actually solved the real-life problems implied by their stories. The examples in this article do not promote or recommend any specific object-level solutions for even those clients’ actual specific problems, let alone universal advice for people in similar situations.

So, if you have the impression that I am recommending, for example, specific ways to deal with career or relationship issues, you are extrapolating something that is not actually here: this article is strictly about how the interaction between counterfactuals and moral judgment interferes with our practical thinking processes, not about...
Shared Cache is Going Away
by jefftk — 2019-11-01T15:10:02.635Z

Browsers historically have had a single HTTP Cache. This meant that
if

www.a.example and

www.b.example both used

cdn.example/jquery-1.2.1.js then JQuery would only be
downloaded once. Since it's the same resource regardless of which
site initiates the download, a single shared cache is more
efficient. [1]

Unfortunately, a shared cache enables a privacy leak. Summary of the
simplest version:

I want to know if you're a moderator on www.forum.example.

I know that only pages under
www.forum.example/moderators/private/ load
www.forum.example/moderators/header.css.

When you visit my page I load
www.forum.example/moderators/header.css and see if it came
from cache.

Versions of this have been around for a while, but in March 2019
Eduardo Vela disclosed a

way
to make it much more powerful and reliable. Browsers are
responding by partitioning the cache (

Chrome,

Firefox;
Safari

already
had). [2] It's not clear from me reading the bugs when it will
launch, but it does sound soon. [3]

What does this mean for developers? The main thing is that there's no
longer any advantage to trying to use the same URLs as other sites.
You won't get performance benefits from using a canonical URL over
hosting on your own site (unless they're on a CDN and you're not) and
you have no reason to use the same version as everyone else (but
staying current is still a good idea).

I'm sad about this change from a general web performance perspective
and from the perspective of someone who really likes small independent
sites, but I don't see a way to get the performance benefits without
the leaks.

[1] When I worked on mod_pagespeed, rewriting web
pages so they would load faster, we had an opt-in feature to Canonicalize
JavaScript Libraries.

[2] I was curious if this had launched yet so I made a pair of test pages
and tried it out in WebPageTest for Chrome
Canary and Firefox
Nightly but it's not out yet. I used a WPT script consisting of:

navigate https://www.trycontra.com/test/cache-partition
navigate https://www.bidadance.org/test/cache-partition

[3] Firefox's bug is
marked "fixed" and "Milestone: mozilla70", but I have Firefox 70.0.1
and it doesn't seem to be enabled:

Perhaps this is just the code change and they still need a flag flip?
I don't know how Firefox does this.

Comment via: facebook, lesswrong
Breaking Group Rock Paper Scissors
by jefftk — 2019-10-31T23:50:01.568Z

Let's say you're at a conference and the leader gets everyone to play
group Rock, Paper, Scissors as an icebreaker:

Play Rock, Paper, Scissors in pairs. Best 2 out of 3 wins.

Each time someone loses, they and anyone in their cheering squad
become part of the winner's cheering squad.

Continue until there's a champion.

How do you maximize your chances of winning?

This is structured as a single-elimination tournament, except there's
nothing to ensure that it's a proper binary tree. Because different
games will take different amounts of time, and you're about 50-50 on
winning each match, you want to play as few matches as possible while
your opponents play as many as possible and eliminate each other.
It's like a Slow
Bicycle Race, where everyone has to keep making forward progress,
but only just barely.

(You can also try to win by predicting your opponent's choices, but
that's very hard. Unless your opponent is a small child in which case
an "always choose paper" strategy can work well.)

Comment via: facebook
Doing things in a populated world
by KatjaGrace — 2019-10-31T22:40:01.141Z

The world has lots of people and things in it. And they are organized in such a mishmash that changing a thing will often make large numbers of people better off or worse off. And for a big thing—even a very good big thing—the number who are worse off is very unlikely to be zero.

This means that if you want to do big things, you will either have to make some people worse off, or rearrange the gains to make everyone better off.

If there are only a small number of people involved, you might be able to make everyone better off with a careful choice of things to change. But if the group is large, you will probably need some sort of generic value fluid, that can flow between the parties and fill in the holes such as to make everyone a bit better off, instead of some people much better off and some people worse off. Money and social respect both fill this role, assuming that there aren’t other impediments to using them, but a giant barrel of compensatory apricots might also work.

This suggests that whether big changes are made depends on the availability of workable value fluid, along with the propensity of the powerful to make the less powerful worse off without compensation. The availability of workable value fluid might for instance change according to social or technical technology for maintaining it, as well as impediments to using that technology.

For instance, if a large group of people were already headed to restaurant A, but the group would on net prefer restaurant B, they might not make this switch, because someone who prefers B would have to raise the issue, and it would feel a bit too much like conflict (and annoyance of extra negotiation for everyone). However if a couple of the people who prefer B actually own B and can offer drinks on the house to the group—and that is enough for everyone to prefer B, including the B owners—the switch can happen more easily. (I’m really thinking of things like shifts in legislation or giant infrastructure projects, but much more of my own experience is with groups going to restaurants.)

Is this right? Is it a big factor? (Theoretically salient mechanisms can be pretty minor in the real world.)
The Simulation Epiphany Problem
by Koen.Holtman — 2019-10-31T22:12:51.323Z

Epistemic status: AGI safety story followed by contradictory alternative conclusions. My thoughts about this problem are still in flux. I somewhat randomly decided write it up and post it on Halloween, because the story can be read as a story about a ghost.

My main question is: has this problem been discussed or resolved somewhere already? Pointers or thoughts?

This problem came up in some provable AGI safety work I am doing, where I am trying to find ways to drive a wedge between the implicit axiomatic structure in many AGI agent models, and our understanding of reality. Here is the story.

Dave's house has a robot butler called PAL. PAL is an intelligent agent in the shape of a little cart with a robot arm on top. Dave has just asked PAL to get him a cup of coffee. There are two routes that PAL can take to the coffee machine. The best one is the short route A, but there is also a longer route B.

PAL contains a computer that decides on the next action it should take by using a predictive world model. PAL considers different actions, running simulations to calculate the effect of each. A utility function is applied to the simulation outcomes to find the best action. For safety reasons, PAL's world model also predicts what Dave will do: this helps PAL to avoid colliding with Dave. Just before it was asked to fetch the coffee, PAL has improved its world model, allowing it to simulate Dave much better than ever before.

So Dave has just asked PAL to get him a cup of coffee. Dave is used to seeing PAL take route A to the coffee machine, and is initially puzzled because PAL is driving along route B. But then Dave has an epiphany. Dave knows with very high certainty that no PAL computer has ever made a mistake, so he can conclude with equally high certainty that he is no longer Dave. He is [Dave], a simulated version of Dave created inside PAL while it is computing the utility of taking route B. [Dave] has empirical proof that he is living in a simulation! This simulation epiphany fills [Dave] with a new sense of meaning: he not only knows his creator, he also knows the goals of his creator, and he knows how he can help. [Dave] picks up PAL and runs with it to the coffee machine.

Back in the real world, PAL has just calculated that taking route B will bring it to the coffee machine fastest. Dave sees PAL driving along route B, and wonders what ha...
Tables of Presence, Nearby Absence, & Degrees of Intensity (Novum Organum Book 2: 10-14)
by Francis Bacon — 2019-10-31T21:56:45.934Z

This is the tenth post in the Novum Organum sequence. For context, see the sequence introduction. For the reading guide, see earlier posts in the sequence.

We have used Francis Bacon's Novum Organum in the version presented at www.earlymoderntexts.com. Translated by and copyright to Jonathan Bennett. Prepared for LessWrong by Ruby.

[[Bacon has begun demonstrating his scientific method in earnest. He uses the investigation of the nature of heat as his example. He explains that his approach works by compiling a natural history, i.e., data related to phenomena of interested which he arranges into three tables:

1) A table of presence which lists many examples where phenomena of interest in presence, e.g. many examples of things where we have heat. Bacon is quite thorough.

2) A table of nearby essence. To discriminate the true cause of heat, Bacon looks for examples of things that resemble those in the table of presence yet are lacking the heat. For the example, the light of the moon (cold) is contrasted with the light of the sun (hot) which is interesting given they are both heavenly bodies.

3) A table of degrees or comparison where are examples are brought where the amount of perceived heat differs in degree between things. This is also useful in discriminating the true underlying cause and and nature of heat.

Bacon is thorough in his examples here, listing an extreme variety of things. To his credit, he eventually hits upon that heat is about motion.

One the hand, these passages can be see as boring, but I see them as quite fascinating– it's remarkable to see how on the earlier pioneers of modern science thought and approached solving what to him was a great mystery. He did not rush, he did not get bored, instead he painstakingly collected his data. A superb demonstration of empiricism.]]

Aphorism Concerning the Interpretation of Nature: Book 2: 10-14

10. Having looked at doctrines, we must go on to precepts, dealing with them in the most direct way and not getting things the wrong way around. Guides for the interpretation of nature are of two fundamental kinds: (1) how to draw or fetch up axioms from experience; (2) how to get from axioms to new experiments. Precepts of kind (1) divide into three kinds of service: (i) catering to the senses, (ii) catering to the memory, and (iii) catering to the mind or reason. (i) First of all we must prepare an adequate and sound na...
Aella on Rationality and the Void
by Jacob Falkovich — 2019-10-31T21:40:52.042Z

Cross-posted from Putanumonit.

This is a transcript of a discussion, my speech is in bold and Aella's is in normal text. This is the second part of the interview, part 1 covered Aella's life and background and is found here.

Let’s talk about trauma, which is a concept about which people have very different beliefs and intuitions. I often struggle to understand what people mean by “trauma”, it doesn’t quite map to anything in my experience. For example, a common trope in therapy is that growth and healing only occurs when you surface and face your trauma. This gets applied even to things like “the trauma of your birth” in practices like holotropic breathwork. Do you know what they’re talking about?

I assume it’s something bad that happens to you and makes you view the world as particularly dangerous. But if you look at what happened to you and you can handle that memory, then the rest of the world stops being dangerous and also becomes something you can handle.

I’m not sure this applies to birth, though. I’m open to birth being traumatic or issue-laden, but it doesn’t feel like a framework I’m particularly drawn to.

You wrote a powerful post about trauma being a story and a narrative. When you lived a narrative of grievance and victimhood, feeling that the universe was deformed by a great injustice, that was traumatic for you.

Yes, the sense that the universe needs to be fixed.

It makes a lot of sense that adopting this narrative of injustice causes intense misery. I think it’s a good model of trauma, and it was very well put after you’ve been writing around the subject for a few years. But when it comes to how you should escape this narrative, I think that we strongly disagree.

Your suggestion is:

The trick to healing from suffering, I think, is deciding that the pain was worth it.
How do you decide the pain was worth it? Find out what it gave you. […]
And in deciding that it had been worth it, the suffering ended. Life had no longer cheated her. Life had made a fair trade.

So you switch to a narrative where the bad thing was worth it because of something good it gave you. I understand why this helps with the suffering, but at the cost of convincing yourself of something false. It’s the just-world fallacy.

As an alternative, I think that saying ...