trentmkelly/LessWrong-43k huggingface dataset:
Writing children's picture books
by jessicata — 2019-06-25T21:43:45.578Z
Link: https://unstableontology.com/2019/06/25/writing-childrens-picture-books/

[the text of the post is pasted here, for redundancy]

Here’s an exercise for explaining and refining your opinions about some domain, X:

Imagine writing a 10-20 page children’s picture book about topic X. Be fully honest and don’t hide things (assume the child can handle being told the truth, including being told non-standard or controversial facts).

Here’s a dialogue, meant to illustrate how this could work:

A: What do you think about global warming?

B: Uhh…. I don’t know, it seems real?

A: How would you write a 10-20 page children’s picture book about global warming?

B: Oh, I’d have a diagram showing carbon dioxide exiting factories and cars, floating up in the atmosphere, and staying there. Then I’d have a picture of sunlight coming through the atmosphere, bounding off the earth, then going back up, but getting blocked by the carbon dioxide, so it goes back to the earth and warms up the earth a second time. Oh, wait, if the carbon dioxide prevents the sunlight from bouncing from the earth to the sky, wouldn’t it also prevent the sunlight from entering the atmosphere in the first place? Oh, I should look that up later [NOTE: the answer is that CO2 blocks thermal radiation much more than it blocks sunlight].

Anyway, after that I’d have some diagrams showing global average temperature versus global CO2 level that show how the average temperature is tracking CO2 concentration, with some lag time. Then I’d have some quotes about scientists and information about the results of surveys. I’d show a graph showing how much the temperature would increase under different conditions… I think I’ve heard that, with substantial mitigation effort, the temperature difference might be 2 degrees Celsius from now until the end of the century [NOTE: it's actually 2 degrees from pre-industrial times till the end of the century, which is about 1 degree from now]. And I’d want to show what 2 degrees Celsius means, in terms of, say, a fraction of the difference between winter and summer.

I’d also want to explain the issue of sea level rise, by showing a diagram of a glacier melting. Ice floats, so if the glacier is free-floating, then it melting doesn’t cause a sea level rise (there’s some scientific principle that says this, I don’t remember what it’s called), but if the glacier is on land, then when it melts, it causes the sea level to rise. I’d also want to show a map of the areas t

...
Only optimize to 95 %
by Leo Shine — 2019-06-25T20:50:27.908Z

I was reading Tom Chivers book "The AI does not hate you" and in a discussion about avoiding bad side effects when asking a magic broomstick to fill a water bucket, it was suggested that somehow instead of asking the broomstick to fill the bucket you could do something like ask it to become 95 percent sure that it was full, and that might make it less likely to flood the house.

Apparently Tom asked Eliezer at the time and he said there was no known problem with that solution.

Are there any posts on this? Is the reason why we don't know this won't work just because it's hard to make this precise?
Research Agenda in reverse: what *would* a solution look like?
by Stuart_Armstrong — 2019-06-25T13:52:48.934Z

I constructed my AI alignment research agenda piece by piece, stumbling around in the dark and going down many false and true avenues.

But now it is increasingly starting to feel natural to me, and indeed, somewhat inevitable.

What do I mean with that? Well, let's look at the problem in reverse. Suppose we had an AI that was aligned with human values/preferences. How would you expect that to have been developed? I see four natural paths:

Effective proxy methods. For example, Paul's amplification and distillation, or variants of revealed preferences, or a similar approach. The point of this that it reaches alignment without defining what a preference fundamentally is; instead it uses some proxy for the preference to do the job.

Corrigibility: the AI is safe and corrigible, and along with active human guidance, manages to reach a tolerable outcome.

Something new: a bold new method that works, for reasons we haven't thought of today (this includes most strains of moral realism).

An actual grounded definition of human preferences.

So, if we focus on scenario 4, we need a few things. We need a fundamental definition of what a human preference is (since we know this can't be defined purely from behaviour). We need a method of combining contradictory and underdefined human preferences. We also need a method for taking into account human meta-preferences. And both these methods has to actually reach an output, and not get caught in loops.

If those are the requirements, then it's obvious why we need most of the elements of my research agenda, or something similar. We don't need the exact methods sketched out there, there may be other way of synthesising preferences and meta-preferences together. But the overall structure - a way of defining preferences, and ways of combining them that produce an output - seems, in retrospect, inevitable. The rest is, to some extent, just implementation details.
Arbital Scrape V2
by emmab — 2019-06-25T10:03:21.962Z

I've scraped http://arbital.com as the site is unusably slow and hard to search for me.

https://drive.google.com/open?id=1b7dKhOzfMpFwngAeI8efeOzv147Lv_mx

https://mega.nz/#!SUQg1YxS!G7bmpNipcl1Ztugvr6l6sUJUu1D0gDg1-jA6oXLNEWg

https://emma-borhanian.github.io/arbital-scrape/

The scrape is locally browsable and plain HTML save for MathJax and a few interactive demos. Source code included (with git history).

(previously Arbital Scrape)

Updates: Included source code, MathJax and link formatting, cross-linking, missing pages, etc

Source: https://github.com/emma-borhanian/arbital-scrape

Mirror: www.obormot.net/arbital
What's up with self-esteem?
by abramdemski — 2019-06-25T03:38:15.991Z

Often, people think about their self-worth/self-confidence/self-esteem/self-efficacy/self-worth in ways which seem really strange from a simplistic decision-theoretic perspective. (I'm going to treat all those terms as synonyms, but, feel free to differentiate between them as you see fit!) Why might you "need confidence" in order to try something, even when it is obviously your best bet? Why might you constantly worry that you're "not good enough" (seemingly no matter how good you become)? Why do people especially suffer from this when they see others who are (in some way) much better than them, even when there is clearly no threat to their personal livelihood? Why might you think about killing yourself due to feeling worthless? (Is there an evo-psych explanation that makes sense, given how contrary it seems to survival of the fittest?)

There might be a lot of diverse explanations for the diverse phenomena. I think providing more examples of puzzling phenomena is an equally valuable way to answer (though maybe those should be a comment rather than an answer?).

This seems connected to the puzzling way people constantly seem to want to believe good things (even contrary to evidence) in order to feel good, and fear failure even when the alternative is not trying & essentially failing automatically.

Some sketchy partial explanations to start with:

Maybe there is a sense in which we manage the news constantly. It could be that we have a mental architecture which looks a lot like a model-free RL agent connected up to a world model, being rewarded for taking actions which increase expected value according to the world-model. The model-free RL will fool the world-model where it can, but this will be ineffective in any case where the world-model understands such manipulation. So things basically even out to rational behavior, but there's always some self-delusion going on at the fringes. (This only has to do with the observation that people sometimes try to make themselves feel better by finding arguments/activities which boost self-esteem, not with other weird aspects of self-esteem.)
There's a theory that, in order to be trustworthy bargaining partners, people evolved to feel guilty/shameful when they violate trust. You can tell who feels more guilt/shame after some interaction with them, and you can expect these people to vio

...
Causal Reality vs Social Reality
by Ruby — 2019-06-24T23:50:19.079Z

Epistemic status: this is a new model for me, certainly rough around the joints, but I think there’s something real here.

This post begins with a confusion. For years, I have been baffled that people, watching their loved ones wither and decay and die, do not clamor in the streets for more and better science. Surely they are aware of the advances in our power over reality in only the last few centuries. They hear of the steady march of technology, Crispr and gene editing and what not. Enough of them must know basic physics and what it allows. How are people so content to suffer and die when the unnecessity of it is so apparent?

It was a failure of my mine that I didn’t take my incomprehension and realize I needed a better model. Luckily, RomeoStevens recently offered me an explanation. He said that most people live in social reality and it is only a minority who live in causal reality. I don’t recall Romeo elaborating much, but I think I saw what he was pointing at. This rest of this post is my attempt to elucidate this distinction.

Causal Reality

Causal reality is the reality of physics. The world is made of particles and fields with lawful relationships governing their interactions. You drop a thing, it falls down. You lose too much blood, you die. You build a solar panel, you can charge your phone. In causal reality, it is the external world which dictates what happens and what is possible.

Causal reality is the reality of mathematics and logic, reason and argument. For these too, it would definitely seem, exist independent of the human minds who grasp them. Believing in the truth preservation of modus ponens is not so different from believing in Newton’s laws.

Necessarily, you must be inhabiting causal reality to do science and engineering.

In causal reality, what makes things good or bad are their effects and how much you like those effects. My coat keeps me warm in the cold winter, so it is a good coat.

All humans inhabit causal reality to some extent or another. We avoid putting our hands in fire not because it is not the done the thing, but because of prediction that it will hurt.

Social Reality

Social reality is the reality of people, i.e. people are the primitive elements rather than particles and fields. The fundamentals of the ontology are beliefs, judgments, roles, relationships, and culture. The most important properties of any object, thing, or idea ...
Machine Learning Projects on IDA
by Owain_Evans — 2019-06-24T18:38:18.873Z

TLDR

We wrote a 20-page document that explains IDA and outlines potential Machine Learning projects about IDA. This post gives an overview of the document.

What is IDA?

Iterated Distillation and Amplification (IDA) is a method for training ML systems to solve challenging tasks. It was introduced by Paul Christiano. IDA is intended for tasks where:

The goal is to outperform humans at the task or to solve instances that are too hard for humans.

It is not feasible to provide demonstrations or reward signals sufficient for super-human performance at the task

Humans have a high-level understanding of how to approach the task and can reliably solve easy instances.

The idea behind IDA is to bootstrap using an approach similar to AlphaZero, but with a learned model of steps of human reasoning instead of the fixed game simulator.

Our document provides a self-contained technical description of IDA. For broader discussion of IDA and its relevance to value alignment, see Ought's presentation, Christiano's blogpost, and the Debate paper. There is also a technical ML paper applying IDA to algorithmic problems (e.g. shortest path in a graph).

ML Projects on IDA

Our document outlines three Machine Learning projects on IDA. Our goal in outlining these projects is to generate discussion and encourage research on IDA. We are not (as of June 2019) working on these projects, but we are interested in collaboration. The project descriptions are “high-level” and leave many choices undetermined. If you took on a project, part of the work would be refining the project and fixing a concrete objective, dataset and model.

Project 1: Amplifying Mathematical Reasoning

This project is about applying IDA to problems in mathematics. This would involve learning to solve math problems by breaking them down into easier sub-problems. The problems could be represented in a formal language (as in this paper) or in natural language. We discuss a recent dataset of high-school problems in natural language, which was introduced in this paper. Here are some examples from the dataset:

Question:
Let u(n) = -n^3 - n^2.
Let e(c) = -2*c^3 + c.
Let f(j) = -118*e(j) + 54*u(j).
What is the derivative of f(a)?

Answer: 546*a^2 - 108*a - 118

Question: Three letters picked without replacement from
qqqkkklkqkkk. Give probability of sequence qql.

Answer: 1/110

The paper showed impressive results on the dataset

...
Being the (Pareto) Best in the World
by johnswentworth — 2019-06-24T18:36:45.929Z

The generalized efficient markets (GEM) principle says, roughly, that things which would give you a big windfall of money and/or status, will not be easy. If such an opportunity were available, someone else would have already taken it. You will never find a $100 bill on the floor of Grand Central Station at rush hour, because someone would have picked it up already.

One way to circumvent GEM is to be the best in the world at some relevant skill. A superhuman with hawk-like eyesight and the speed of the Flash might very well be able to snag $100 bills off the floor of Grand Central. More realistically, even though financial markets are the ur-example of efficiency, a handful of firms do make impressive amounts of money by being faster than anyone else in their market. I’m unlikely to ever find a proof of the Riemann Hypothesis, but Terry Tao might. Etc.

But being the best in the world, in a sense sufficient to circumvent GEM, is not as hard as it might seem at first glance (though that doesn’t exactly make it easy). The trick is to exploit dimensionality.

Consider: becoming one of the world’s top experts in proteomics is hard. Becoming one of the world’s top experts in macroeconomic modelling is hard. But how hard is it to become sufficiently expert in proteomics and macroeconomic modelling that nobody is better than you at both simultaneously? In other words, how hard is it to reach the Pareto frontier?

Having reached that Pareto frontier, you will have circumvented the GEM: you will be the single best-qualified person in the world for (some) problems which apply macroeconomic modelling to proteomic data. You will have a realistic shot at a big money/status windfall, with relatively little effort.

(Obviously we’re oversimplifying a lot by putting things like “macroeconomic modelling skill” on a single axis, and breaking it out onto multiple axes would strengthen the main point of this post. On the other hand, it would complicate the explanation; I’m keeping it simple for now.)

Let’s dig into a few details of this approach…

Elbow Room

There are many table tennis players, but only one best player in the world. This is a side effect of ranking people on one dimension: there’s only going to be one point furthest to the right (absent a tie).

Pareto optimality pushes us into more dimensions. There’s only one best table tennis player, and only one best 100-meter sprinter, b...
[AN #58] Mesa optimization: what it is, and why we should care
by Rohin Shah — 2019-06-24T16:10:01.330Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Highlights

Risks from Learned Optimization in Advanced Machine Learning Systems (Evan Hubinger et al): Suppose you search over a space of programs, looking for one that plays TicTacToe well. Initially, you might find some good heuristics, e.g. go for the center square, if you have two along a row then place the third one, etc. But eventually you might find the minimax algorithm, which plays optimally by searching for the best action to take. Notably, your outer optimization over the space of programs found a program that was itself an optimizer that searches over possible moves. In the language of this paper, the minimax algorithm is a mesa optimizer: an optimizer that is found autonomously by a base optimizer, in this case the search over programs.

Why is this relevant to AI? Well, gradient descent is an optimization algorithm that searches over the space of neural net parameters to find a set that performs well on some objective. It seems plausible that the same thing could occur: gradient descent could find a model that is itself performing optimization. That model would then be a mesa optimizer, and the objective that it optimizes is the mesa objective. Note that while the mesa objective should lead to similar behavior as the base objective on the training distribution, it need not do so off distribution. This means the mesa objective is pseudo aligned; if it also leads to similar behavior off distribution it is robustly aligned.

A central worry with AI alignment is that if powerful AI agents optimize the wrong objective, it could lead to catastrophic outcomes for humanity. With the possibility of mesa optimizers, this worry is doubled: we need to ensure both that the base objective is aligned with humans (called outer alignment) and that the mesa objective is aligned with the base objective (called inner alignment). A particularly worrying aspect is deceptive alignment: the mesa optimizer has a long-term mesa objective, but knows that it is being optimized for a base objective. So, it optimizes the base objective during training to avoid being modified, but at deployment when the threat of modification is ...
How does one get invited to the alignment forum?
by Chris_Leong — 2019-06-23T09:39:20.042Z

At the moment there is a link on the Alignment Forum to apply for membership. You can apply by submitting papers, blog posts or comments. However, there is very little in the way of detail of what kind of work they expect they expect in order for you to have a reasonable chance of being invited. I'm not saying that there should be specific criteria as I'm in favour of the moderators using their judgement. However, more detail on what they are looking for would provide encouragement for people to work towards achieving the level of knowledge and good judgement that they expect for their members.
Podcast - Putanumonit on The Switch 
by Jacob Falkovich — 2019-06-23T04:09:25.723Z

I’m really excited to make my podcast debut on The Switch podcast with Chase Harris and Alex Berner. Here are the links on Apple Podcasts, Play Music, and Spotify.

Topics we covered:

How business school led me to rationality.
The origin story and meaning of Putanumonit.
How to put numbers on things where no numbers exist.
Contextualizing and decoupling in the saga of Caster Semenya.
Does putting numbers on dating make me an emotionless robot?
Rationality, mindfulness, and poking your head above the river.
The posts I regret writing.
Antinatalism and the connection between emotion and philosophy.
How intuition follows controversy, and why hunter-gatherers don’t have opinions on immigration.
Fake frameworks as the key to rationality and why I prefer Magic the Gathering personality color wheel to the big 5 personality system or MBTI.
Rationality alone and in a group.
Why soccer is a supreme entertainment product, aesthetic experience, and showcase of virtue.
MMA as a gateway drug to loving sports.
The Foundational Toolbox for Life: Introduction
by ExCeph — 2019-06-22T06:11:59.497Z

(Co-authored with Sailor Vulcan)

Follows from:

What Do We Mean by Rationality?

No Really, Why Aren't Rationalists Winning?

Related resources:

Inadequate Equilibria

Instrumental Rationality sequence

How the ‘Magic: The Gathering’ Color Wheel Explains Humanity

What’s Going On Here?

As I was growing up, I could tell that something critical was missing from many of the adults that I encountered, and from the way society as a whole functioned. People regularly made decisions that seemed obviously stupid, and the institutions they built were little better. When I eventually learned of others who agreed that society was doing a poor job of addressing problems, I was relieved, but still puzzled that nobody had managed to fix the situation yet.

As I learned the basics of rationality, I became more confused. People had created formalized methods for making better decisions based on understanding how systems and logic worked, which was amazing. However, not only had these methods failed to become the default standard of human adulthood, not only were they not used to guide policy decisions in the public and private sectors, but the methods were obscure. Barely anybody knew they existed. The world rationalists said they wanted was nowhere in sight.

I concluded that there had to be more to implementing effective policies and institutions than just knowing how things worked. If I wanted to solve any important problems like that, I would have to develop other skills. Since all I had was the proverbial hammer of understanding ideas and mechanisms, I embarked on a journey to hammer out an understanding of skills. I asked myself, what do I need to do to build the world I expect?

A look at the people who seemed to be the ones building the world prompted further questions: Why are non-rationalists having so much more impact than rationalists? What skills are they applying that rationalists aren’t? And how do these non-rationalists successfully learn and apply skills they don't fully understand?

Cognitive Filters; or, “Parallel Realities of a Plane”

As was pointed out in

What Do I mean by Rationality

, it’s not possible to form an accurate view of the world in any reasonable amount of time by tracking minute observations one by one and plugging the data from that into Bayesian formalisms in their full form. This is a principle of epistemic rationality.

I realized there must als...