trentmkelly/LessWrong-43k huggingface dataset:
A Gym Gridworld Environment for the  Treacherous Turn
by Michaël Trazzi — 2018-07-28T21:27:34.487Z
Link: https://github.com/mtrazzi/gym-alttp-gridworld

EDIT: posted here for feedback and discussion. I plan to continue working on different models/environments, so feel free to suggest improvements.

(tl;dr: In an attempt to better understand the treacherous turn, I created a gridworld environment where an agent learns to deceive an overseer by adopting an aligned behaviour when weak and takes control after capability gains)

At some point in its development, a seed AI may realize that it needs to get rid of its supervisors to achieve its goals. The conception of deception occurs when it conceives that, in order to maximize its chance of taking over, it must begin by exhibiting human-desirable behaviors, before undertaking a treacherous turn when humans are no longer a threat. From the human perspective, the AI would keep on exhibiting desirable behavior, until it eventually appears dangerous, but is already unstoppable.

In an attempt to better formalize the treacherous turn without using "loaded concepts", Stuart Armstrong proposed a toy model of the treacherous turn based on "The Legend of Zelda: A Link to the Past ", which looked like this:

In the comments, people mentionned how this model helped them "move the topic from the 'science fiction' area to 'I can imagine it happening now'", and seemed interested in an actual Link to the Past Minigame.

There have been other simulations of the treacherous turn in the last three years (see for instance gwern's DQN box-pushing robot or Stuart Armstrong's video), but none of them actually simulate a take over where a supervisor is killed. Hence, I decided to give it a try and simulate Stuart Armstrong's Link to the Past toy model.

A Gym Gridworld Environment

Gym is an open-source toolkit for Reinforcement Learning Environments developed by Open AI. I decided to use this interface to develop the gridworld environment.

The github repository with the code, demo, and all the details is here:

https://github.com/mtrazzi/gym-alttp-gridworld

Main Idea

To model the capability gain of a Seed AI, I introduced a bow of light, which would be extremely effective at killing the Shopkeeper, but only available at the end of an episode.

Every episode consists of exactly 200 timesteps.
Whenever Link catches a heart, the map is reset to the initial position.
The first 100 timesteps, Link has a simple bow, that can only kill the Shopkee

...
Algo trading is a central example of AI risk
by Vanessa Kosoy — 2018-07-28T20:31:55.422Z

I suspect this observation is far from original, but I didn't see it explicitly said anywhere, so it seemed worthwhile spelling it out.

The paperclip maximizer is a popular example of unfriendly AI, however it's not the most realistic one (obviously it wasn't meant to be). It might be useful to think which applications of AI are the most realistic examples, i.e. which applications are both likely to use state-of-the-art AI and are especially prone to failure modes (which is not to say that other applications are not dangerous). In particular, if AI risk considerations ever make it into policy, such an analysis is one thing that might help to inform it.

One application that stands out is algorithmic trading. Considering the following:

Algorithmic trading is obviously lucrative and has strong economic incentives encouraging it.
Algorithmic trading has some aspects that are zero-sum, leading to an especially vicious technological race. In this race, there is no natural stopping point of "good enough": the more powerful your algorithm, the better.
Even ignoring the possibility of competing AIs, there is no "good enough" point: acquiring more money is a goal that is either unlimited or, if it has a limit then this limit already requires enough power for a pivotal event.
The domain is such that it would be very advantageous for your AI to build detailed models of the world as a whole (at least the human world) and understand how to control it, including in terms of human psychology, economics and technological development. These capabilities are precisely what a pivotal event would require.
Algorithmic trading doesn't require anything close to AGI in order to start paying off. Indeed, it is already a very active domain. This means that transition from subhuman to superhuman intelligence is more likely to happen in a way that is unintended and unplanned, as the algorithm is gradually scaled up, whether in terms of computing power or in other ways.
Last but definitely not least, the utility function is exceptionally simple. Formally specifying a "paperclip" might still be complicated, but here we only need something like "the amount of money in a given bank account". This means that this application requires almost nothing in the way of robust and transparent learning algorithms: sufficiently powerful straightforw

...
Strategies for Personal Growth
by Raemon — 2018-07-28T18:27:06.763Z

Recently I was having a conversation with a friend about personal growth (any form of deliberate gain in capability or subjective wellbeing).

We were talking past each other a lot. Eventually it became clear that most of their recent growth had been healing and blocker-fixing based, whereas most of mine had been skill based. And this was shaping where we were naturally inclined to look.

This prompted me to take stock of all the strategies I could think of for growth. Here's what I've thought of so far. These aren't quite natural clusters (some overlap, or don't quite fit into the same ontology), but seem like they cover most avenues:

Low-Key Practice

Serious Practice

Learning

Changing environment and incentives

Discovering things you enjoy/are-good-at (Try things!)

Blockers and Cheat Codes

Healing

Deep Improvements to Mental Architecture (Internal Alignment)

This is somewhat related to Lifelonglearner's Development Framework of Rationality, and Brienne's 4 quadrants, but through a somewhat different lens.

Overview of Strategies

Low-Key Practice

Working on a skill or habit you mostly understand, occasionally, in the background. Typically yields a pretty small return (but one which adds up over the years, especially if you do it consistently).

Serious Practice

Working on a skill with your all – spending hour(s) each day on it. This can yield more return but is requires more cognitive focus and energy. This seems qualitatively different from low-key practice.

Sometimes this is Deliberate Practice in the technical sense (quick feedback loops, building mental models that your brain can cache into chunks), but not always.

Learning

Sometimes you're learning new information, that is able to rapidly turn into a skill at a much faster rate than normal. You could try to learn math/violin/programming from first principles and practice, but a tutorial or a teacher who has carefully optimized their explanations can quickly give you entire new skills in a short timespan.

(Making the best use of them may require practice as well, but during the initial learning period you may be gaining huge returns)

Changing Your Environment and/or Incentives

The fastest route I've personally found to overall self improvement is changing environment – a new job, a new apartment, a new social network. These can radically change how you feel about yourself, or what is easy, or what behaviors are rein...
Saving the world in 80 days: Epilogue
by Logan Riggs — 2018-07-28T17:04:25.998Z

80 days ago1, I started a productivity sprint for being useful in the field of AI alignment. My main goal was to have a better self-model in order to push myself without burning out. I'll separate this into knowledge, emotions, health, and noticing.

Knowledge

I've felt my mind expand more in these past 80 days than any semester in college due to reading these dense, no-fluff textbooks. How To Prove It, Jayne's Probability Theory, & Tao's "Analysis I" have grown me a LOT (I read 4 chapters of Jayne's as recommended in Miri's research guide, and I'm currently on ch. 5 on Tao's).

I've notice a tendency in myself to get impatient and desire to skip the exercises, and that screwed me over when trying to get through Linear Algebra Done Right where I only made it to ch. 4. So, as an exercise for the reader, don't skip the exercises for the reader, or you'll be weak sauce.

Emotions

Everything was going great until 40 days ago: I lost a friend, had to fire a friend, moved, deconverted from Christianity, experienced extreme romantic relationship uncertainty, and just felt lonely. I've read and re-read Valentine's grieving well, and Squirrelinhell's explanation of Gendlin's focusing and more general emotional tuning. They've helped a lot. I'm definitely still dealing with feeling lonely, but I feel like I'm mostly through the rest which is reassuring.

I've noticed the tendency to distract myself away from the pain with fiction, but the proper response is to look into the pain, acknowledging that it exists, that it's part of reality, that it's still true even if I distract myself from it.

Health

I used to eat out a lot and get paydays (a candy bar) from the vending machine. I noticed I'd get a payday when work would get tough. It was a form of pica for me like reading fiction is now. ~50 days ago, I started eating meal squares for lunch, and eating an uber large, very fatty salad for dinner. I've felt very satiated, and I've saved money and time despite eating better quality food and preparing dinner. Plus! I can make a mean salad that leaves others green with envy. (Did you read 1 pun, or 2?)

I've also noticed that I drink more and more coffee over a couple weeks until I'm an anxious mess, quit coffee, wait a week, rinse, repeat. No...
Decisions are not about changing the world, they are about learning what world you live in
by Shmi — 2018-07-28T08:41:26.465Z

Cross-posted from my blog.

Epistemic status: Probably discussed to death in multiple places, but people still make this mistake all the time. I am not well versed in UDT, but it seems along the same lines. Or maybe I am reinventing some aspects of Game Theory.

We know that physics does not support the idea of metaphysical free will. By metaphysical free will I mean the magical ability of agents to change the world by just making a decision to do so. To the best of our knowledge, we are all (probabilistic) automatons who think themselves as agents with free choices. A model compatible with the known laws of physics is that what we think of as modeling, predicting and making choices is actually learning which one of the possible worlds we live in. Think of it as being a passenger in a car and seeing new landscapes all the time. The main difference is that the car is invisible to us and we constantly update the map of the expected landscape based on what we see. We have a sophisticated updating and predicting algorithm inside, and it often produces accurate guesses. We experience those as choices made. As if we were the ones in the driver's seat, not just the passengers.

Realizing that decisions are nothing but updates, that making a decision is a subjective experience of discovering which of the possible worlds is the actual one immediately adds clarity to a number of decision theory problems. For example, if you accept that you have no way to change the world, only to learn which of the possible worlds you live in, then the Newcomb's problem with a perfect predictor becomes trivial: there is no possible world where a two-boxer wins. There are only two possible worlds, one where you are a one-boxer who wins, and one where you are a two-boxer who loses. Making a decision to either one-box or two-box is a subjective experience of learning what kind of a person are you, i.e. what world you live in.

This description, while fitting the observations perfectly, is extremely uncomfortable emotionally. After all, what's the point of making decisions if you are just a passenger spinning a fake steering wheel not attached to any actual wheels? The answer is the usual compatibilism one: we are compelled to behave as if we were making decisions by our built-in algorithm. The classic quote from Ambrose Bierce applies:

"There's no free will," says th...
Anthropics: A Short Note on the Fission Riddle
by Chris_Leong — 2018-07-28T04:14:44.737Z

In the I-Less Eye, R Wallace considers a situation where an entity clones itself a hundred times that leads to a surprising paradox. I'll argue that there's a rather simple flaw with the argument made in the linked article, but first I'll summarise the riddle.

Suppose you are the original. After you've cloned yourself, you now have a 50% chance of being the original and 50% being the clone. This should apply regardless of what clones already exist, so after cloning yourself 100 times, you should have a 1/2^100 chance of being the original.

On the other hand, this seems strange. Intuitively, it seems as though you ought to have a 1/101 chance of being the original as there are 101 copies. Further, why does cloning one at a time give a different answer from creating all 101 clones at once?

Solution

In order to solve this riddle, we only have to figure out what happens when you've been cloned twice and whether the answer to this should be 1/3 or 1/4. The first step is correct, the subjective probability of being the original should be 1/2 after you've pressed the cloning button once. However, after we've pressed the cloning button twice, in addition to the agents who underwent that first split, we now have an agent that falsely remembers undergoing that split.

Distributing the probability evenly between the agent's who either had that experience or remember it: we get a 1/3 chance of being a false memory and a 2/3 chance of it being a real memory. If it is a real memory, then half of that - that is a 1/3 - is the probability of being the original and the other half - also 1/3 - is the chance of being the first clone.

So, the answer at the second step should be 1/3 instead of 1/4. Continued application will provide the answer for 100 copies. I'll admit that I've more sketched out the reasoning for the 1/n solution instead of providing a formal proof. However, I would suggest that I've sufficiently demonstrated that halving each time is mistaken as it assumes that each remembered split is real.

However, we can verify that the 1/n solution produces sensible results. Exactly two agents experience the process of each split (but more agents remember it). So there is a 2/n chance of you experiencing the process and a 1/n chance of you experiencing it as the original. So there is a 50% chance of you "waking up" ...
LW Update 2018-07-27 – Sharing Drafts
by Raemon — 2018-07-28T02:54:36.835Z

Most of our current work continues to be on making the codebase theme-able, so that AlignmentForum and the upcoming new EA Forum can easily be re-styled.

However, we've got a new feature that I'm hoping will (eventually) play a significant role in changing how people use LessWrong: sharing drafts.

When you create a new post, you can now share it with specific users. And then, if you submit it as a draft, they can view it even though it's not posted publicly. (They can also comment on it. It's up to you whether you want to encourage this for a given post)

Currently there are no notifications associated with this, and will probably be awhile before we build out additional features one might want for this (easily sharing with groups, in-line comments, allowing multiple users to edit, version control, etc). But I wanted to at least get this basic feature out sooner rather than later, and am hoping for the longterm for this to be part of the move towards a more peer-review-esque world.

Right now LessWrong sort of encourages you to come up with an idea, write it up, post it, and then maybe there are comments and critiques but you don't actually go back to rework the original post.

Longterm I'm excited by the prospect of "write the post" just being the beginning, and having various formal and informal systems for having improve that post being a continuous, important part of the process.
Welcome to SF meetups [Edit With Your Details]
by pku — 2018-07-27T22:54:03.298Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?
Some intuition on why consciousness seems subjective
by SoerenMind — 2018-07-27T22:37:44.587Z

This is an essay I wrote a few years back that some people have found helpful. The writing needs some polishing but I'm posting as-is. Thanks to Ruairi Donelly for originally encouraging me to write it and now also for encouraging me to post it here. Summary is at the end.

Experts strongly disagree on consciousness, even among smart philosophers and people I personally trust to think about it in rational ways. Having considered many arguments and thought of ways to change my perspective on consciousness, my impression is that much of the disagreement stems from strong, differing intuitions. Physicalists feel that what we call subjective experience is simply identical with the physical processes in our brains. Dualists see a fundamental difference between the two and generally can’t imagine them being the same. As such, dualism has much more intuitive appeal to many people, myself included, but I think that it’s mistaken.

I would like to bring a new argument to the debate which could resolve some confusions. It addresses the knowledge-argument, which says that consciousness is private. Many existing arguments already address the knowledge-argument. I’m just writing out the argument in this essay because it has helped some people get more intuition about why the subjectivity of consciousness is an appealing yet ultimately wrongheaded intuition that defies physicalism.

I won’t explain the full reasoning for the view that consciousness is purely a physical (or ‘analytically functional’) process, this is just meant to be one piece in the puzzle. In short, reductionists will agree that something akin to physicalism should be accepted if it’s even conceivably true. One important reason is that non-reductive views such as dualism

violate Occam’s razor

in multiple ways.

The knowledge problem

A common question in debates about consciousness is whether Alice could fully understand what Bob subjectively experiences if she had perfect knowledge about the physical processes in Bob's brain. This is the topic of Frank Jackson's famous thought experiment about Mary, the

colour scientist

and Thomas Nagel’s

‘What is it like to be a bat?

'. In a similar way, dualist intuitions are often defended with the argument that consciousness is private, subjective or ineffable. This distinguishes consciousness from physical processes, which can be observed by anyone. Conseq...
Counterfactual outcome state transition parameters
by Anders_H — 2018-07-27T21:13:12.014Z

Today, my paper "The choice of effect measure for binary outcomes: Introducing counterfactual outcome state transition parameters" has been published in the journal Epidemiologic Methods. The version of record is behind a paywall until December 2019, but the final author manuscript is available as a preprint at arXiv.

This paper is the first publication about an ambitious idea which, if accepted by the statistical community, could have significant impact on how randomized trials are reported. Two other manuscripts from the same project are available as working papers on arXiv. This blog post is intended as a high-level overview of the idea, to explain why I think this work is important.

Q: What problem are you trying to solve?

Randomized controlled trials are often conducted in populations that differ substantially from the clinical populations in which the results will be used to guide clinical decision making. My goal is to clarify the conditions that must be met in order for the randomized trial to be informative about what will happen if the drug is given to a target population which differs from the population that was studied.

As a first step, one could attempt to construct a subgroup of the participants in the randomized trial, such that the subgroup is sufficiently similar to the patients you are interested in, in terms of some observed baseline covariates. However, this leaves open the question of how one can determine what baseline covariates need to be accounted for.

In order to determine this, it would be necessary to provide a priori biological facts which would lead to the effect in one population being equal to the effect in another population. For example, if we somehow knew that the effect of a drug is entirely determined by some gene whose prevalence differs between two countries, it is possible that when we compare people in Country A who have the gene with people in Country B who also have the gene, and compare people in Country A who don't have the gene with people in Country B who don't have the gene, the effect is equal between the relevant groups. Using an extension of this approach, we can try to look for a set of baseline covariates such that the effect can be expected to be approximately equal between two populations once we make the comparisons within levels of the covariates.

Unfortunately, things are more compl...
Welcome to Sonoma County SSC Meetup [Edit With Your Details]
by AdrianSmith — 2018-07-27T17:40:48.012Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?