trentmkelly/LessWrong-43k huggingface dataset:
A small example of one-step hypotheticals
by Stuart_Armstrong — 2019-01-28T16:12:02.722Z

Just a small example of what one-step hypotheticals might mean in theory and in practice.

This involves a human H pricing some small object:

In theory

The human H is (hypothetically) asked various questions that causes it to model how much they would pay for the small violin. These questions are asked at various times, and with various phrasings, and the results look like this:

Here the costings are all over the place, and one obvious way of reconciling them would be to take the mean (indicated by the large red square), which is around 5.5.

But it turns out there are extra patterns in the hypotheticals Ht and the answers f(Ht). For example, there is a clear difference between valuations that are done in the morning, around midday, or in the evening. And there is a difference if the violin is (accurately) described as "handmade"L

There are now more options for finding a "true" valuation here. The obvious first step would be to over-weight the evening valuations, as there are less datapoints there (this would bring the average up a bit). Or one could figure out whether the "true" H was better represented by their morning, midday, or evening selves. Or whether their preference for "handmade" objects was strong and genuine, or a passing positive affect. H's various meta-preferences would all be highly relevant to these choices.

In practice

Ok, that's what might happen if the agent had the power to ask unlimited hypothetical questions in arbitrarily many counterfactual scenarios. But that is not the case in the real world: the agent would be able to ask one, or maybe two questions at most, before the human attitude to the violin would change, and further data would become tainted.

Note that if the agent had a good brain model of H, it might be able to simulate all the relevant answers; but we'll assume for the moment that the agent doesn't have the capabilities.

So, in theory, huge amounts of data and many relevant patterns that are meta-preferentially relevant. In practice, two values maximum.

Now, if this was all that the agent had access to, then it could only use a crude guess. But if the agent was investigating the human more thoroughly, it could do a lot more. The pattern of valuing things differently at different times of the day might show up over longer observations, as would the pattern of reacting to key words in the description. If the agent assumed that "va

...
How would one go about defining the ideal personality compatibility test?
by digitalcaffeine — 2019-01-28T03:02:04.777Z

Let's talk "soulmates".

Pop culture has more or less drilled into our minds that the idea of machine-found love is dystopian and scary in nature, as if intoxicatedly hitting on members of the other sex at bars, or vainly swiping on profile pictures could possibly lead a person to their ideal match.

On the other hand, we already do have personality compatibility tests such as the Myers Briggs, not that its 16 personality results serve as anything other than a vague, pseudoscience guideline in regards to whom you would probably get along with. I'd like to think we can do better than this.

So, here's my question for you, LessWrong: Using only practical elements (re: technologies we have available to us) and preferably a reasonable number of questions (<100?), what would you define as a quiz set that would work best in matching two hypercompatible personalities together? Assume you're tasked with this in the hopes of creating romantic couples, and that you have a reasonably large user population to work with.
Solomonoff induction and belief in God
by Berkeley Beetle — 2019-01-28T03:01:04.584Z
Link: https://randalrauser.com/2019/01/solomonoff-induction-and-the-alpha-algorithm/

Here’s an article arguing that Solomonoff induction is compatible with belief in a personal God, as long as one accepts that God is computable (that is, there is an Turing machine that computes everything God does and thinks). You might agree or disagree quite strongly, but it’s probably worth checking out, at least as a thought experiment.
"The Unbiased Map"
by SebastianG  — 2019-01-27T19:08:10.051Z

“Long have we suffered under the tyranny of maps.

Biased maps which show topography, but not population.

Wretched maps which speak of religion, but not languages.

Divisive maps which paint with the color of Party, but not the color of economic conditions.

Dirty maps which show crop yield across the heartland, but neglect Fiber Optic Internet coverage.

Our age calls for better maps, maps free from the bias of these old maps, perfect maps.

Imagine the day of the unbiased map. The map which shows both how to get to the airport via public transit and GDP by county. The holy map demonstrating last year’s rainfall and the distribution of seminaries and rabbinical schools. The ancestral map depicting migration of immigrants and American tribes in 1491.

Don’t give me an atlas which pretends at perfection but hisses red herrings from hydra-heads. I want the real thing, a map which doesn’t end at some arbitrary border whether it be the county line, or the sphere of earth. A map which can show the world as known by the Qing Dynasty, Strabo, Majorcan Jews, and the Aztecs. A map of Elon Musk’s neurons and a map of the solar system.

Today’s maps enlighten as the Brothers Grimm, through a bundle of fairy tales. There are no ethical maps under capitalism, all of them drip with nullification and interposition. None show me the world that should be, none provide directions to Valhalla, all show but some thin surface of Reality. And for Mankind, the surface does not satisfy!”
Prediction Contest 2018: Scores and Retrospective
by jbeshir — 2019-01-27T17:20:15.245Z

Way back in April 2018, I announced a Prediction Contest, in which the person who made the best predictions on a bunch of questions on PredictionBook ahead of a 1st July deadline would win a prize after they all resolved in January 2019, which is now.

It was a bit of an experiment; I had no idea how many people were up for practicing predictions to try to improve their calibration, and decided to throw a little money and time at giving it a try. And in the spirit of reporting negative experimental results: The answer was 3, all of which I greatly appreciate for their participation. I don't regret running the experiment, but I'm going to pass on running a Prediction Contest 2019. I don't think this necessarily rules out trying to practically test and compete in rationality-related areas in other ways later, though.

The Results

Our entrants were bendini, bw, and Ialaithion, and their ranked log scores were:

bw: -9.358221122

Ialaithion: -9.594999615

bendini: -10.0044594

This was sufficiently close that changing a single question's resolution could tip the results, so they were all pretty good. That said, bw came out ahead, and even managed to beat averaging everyone's predictions- if you simply took the average prediction (including non-entrants) as of entry deadline and made that your prediction, you'd have got -9.576568147.

The full calculations for each of the log scores, as well as my own log score and the results of feeding the predictions as of prediction time to a simple model rather than simply averaging them, are in a spreadsheet here.

I'll be in touch with bw to sort out their prize this evening, and thanks to everyone who participated and who helped with finding questions to use for it.
Freely Complying With the Ideal: A Theory of Happiness 
by Solnassant — 2019-01-27T12:28:59.810Z

Epistemic status: somewhat empirical, having gone from badly depressed to astoundingly happy within a year through a lot of different experiences (and no drugs). I don’t know whether what I do that makes me happy is idiosyncratic to me and a subgroup of people like me, or universal.

This post is neither an attempt to predict happiness nor to teach Three Easy Steps To Be Happy™. I try to verbalize what I believe causes me to be so happy in such a consistent way.

Over the last few months, I have repetitively done some pretty difficult things, almost entirely because they were part of a path that 10+years-ideal-me would have taken at my age to become him.

One year ago, I asked myself if there was a version of the world in which I’d want to be alive: the answer was yes. That ideal has changed greatly since then; along with the milestones my preferences evolved. Still, the nature of the purpose that guides me remains the same: I try to walk the path that ideal me would be walking.

I think that taking now the actions that will bring about the ideal you or your ideal world will bear immense happiness into your life. I‘ll lay out my explanation of why that is.

Kahneman talks about the tyranny of the remembering self. We take pictures instead of enjoying the moment; in anticipation we choose not the experiences that will be the nicest but those that will be the most memorable, etc. The problem according to him is that this doesn’t optimize for overall happiness, which includes experiential happiness as well. We don’t receive as much value from fondly remembering memorable events as we lost value choosing them over more agreeable ones (because we overestimate how much time we’ll spend remembering).

Well, guess what you do have to remember all the time, as if you’d uncontrollably pressed “repeat” on the memory player? You.

Even if you aren’t pressing the button, everything is a reminder. The people, of your social status; the opposite sex, of you sexual marketplace value; your tests, job, understanding of the world... of your intelligence; and so on.

I‘ll admit here that it gets a little confusing what’s experiential and what’s remembering. Being, you experience you, how you think, how your self and your thoughts change; but everything around you also reminds you how much you suck if you think you do, how much you rock if you think you do, how much you [whatever you ...
Confessions of an Abstraction Hater
by Martin Sustrik — 2019-01-27T05:50:01.066Z

I've written about the cost of abstraction before. Once you are in the IT industry for couple of decades and once you've read couple of millions lines on legacy code you become healthily suspicious of any kind of abstraction. Not that we can do without abstraction. We need it to be able to write code at all. However, each time you encounter an abstraction in the code that could have been avoided you get a little bit sadder. And some codebases are sadder than Romeo and Juliet and King Lear combined.

Remember reading an unfamiliar codebase the last time? Remember how you've thought that the authors were a bunch of incompetent idiots?

People may argue that this is because legacy stuff is necessarily convoluted, but hey, at that point you were just skimming through the codebase and you weren't understanding it deep enough to tell your typical enterprise legacy monstrosity from a work of an architectural genius. The reason you were annoyed was because you were overwhelmed by the sheer amount of unfamiliar abstraction. (To prove that, consider what was your opinion of the codebase was few months later, after getting familiar with it. It looked much better, no?)

Keep that feeling in mind. Think of it when writing new code. How will a person who doesn't know first thing about this codebase feel when reading it?

The options are not palatable. Either you try to be clever, use abstraction a lot and they'll think you are a moron. Or you get rid of all unnecessary abstraction. You'll make their life much less frustrating but they'll think you are some kind of simpleton. (And they'll probably refactor the code to make it look more clever.)

I want to give a very basic example of the phenomenon.

Imagine that the requirements are that your program does A, B, C, D and E, in that order.

You can do it in the dumbest possible way:

void main() { // Do A. ... // Do B. ... // Do C. ... // Do D. ... // Do E. ... }

Or maybe you notice that B, C and D are kind of related and comprise a logical unit of work:

void foo() { // Do B. ... // Do C. ... // Do D. ... } void main() { // Do A. ... foo(); // Do E. ... }

But C would probably be better off as a stand-alone function. You can imagine a case where somewhene would like to call it from elsewhere:

void bar() { // Do C. ... } void foo() { // Do B. ... bar(); // Do D. ... } void main() { // Do A. ... foo(); // Do E. ... }

Now thi

...
Río Grande: judgment calls
by KatjaGrace — 2019-01-27T03:50:01.102Z

In a particularly bad recent bout of anxiety, I learned what seemed to me to be a new and exciting mental move, which I named ‘making a judgment call’. If you had asked me previously whether I made judgment calls, I would have said ‘yes’ however, so describing what I am talking about here is perhaps somewhat subtle.

When I have to decide about something and the relevant issues are not certain, I think I have usually waited for the world to resolve enough uncertainty to make the decision clear. For instance, for it to become clear that the food is safe to eat or sufficiently likely to be unsafe that it should be thrown out. I mean, I thought and collected evidence, but the procedure was to go through these procedures until the answer was returned. The thing I would have called ‘making a judgment call’ was something like ‘think until it becomes clear that the food is safe’ (then ‘the food is safe’ is your judgment) or perhaps ‘think until it is clear at a higher level that more thinking isn’t worthwhile’. You make a judgment by becoming confident in the expected value calculation. It isn’t necessarily an explicit calculation, but you feel confident enough that one side is right.

But you can also just stop before anything is clear. Before you even have a clear assessment of the relevant uncertainties and expected values, or which heuristics are solidly applicable in this case. Instead of waiting until the best way to act reveals itself to you, you can make a decision. You can just say ‘nah, the food is fine, I judged it so’.

Something like that felt like a mental motion that I didn’t know that I could do. A bit like learning to wiggle your ears, when you didn’t know where the muscles were. A friend asked me what this mental motion felt like. I can’t remember what I said, but now I’d say it has a sense of ownership and mineness. I suppose because what was more a feature of the circumstance has been replaced by my own will.

Surely I have always often done something like this in other kinds of cases, e.g. when I’m deciding where to put the tomato on my lunch plate I don’t do any kind of implicit EV calculation that I’m aware of. But the mental motion there feels different—the situation doesn’t present itself as a choice in the same way perhaps. I think I just follow some feeling of what is right. Which seems like a different interesting avenue of decision making expl...
Building up to an Internal Family Systems model
by Kaj_Sotala — 2019-01-26T12:25:11.162Z

Introduction

Internal Family Systems (IFS)

is a psychotherapy school/technique/model which lends itself particularly well for being used alone or with a peer. For years, I had noticed that many of the kinds of people who put in a lot of work into developing their emotional and communication skills, some within the rationalist community and some outside it, kept mentioning IFS.

So I looked at the

Wikipedia page about the IFS model

, and bounced off, since it sounded like nonsense to me. Then someone brought it up again, and I thought that maybe I should reconsider. So I looked at the WP page again, thought “nah, still nonsense”, and continued to ignore it.

This continued until I participated in CFAR mentorship training last September, and we had a class on CFAR’s

Internal Double Crux

(IDC) technique. IDC clicked really well for me, so I started using it a lot and also facilitating it to some friends. However, once we started using it on more emotional issues (as opposed to just things with empirical facts pointing in different directions), we started running into some weird things, which it felt like IDC couldn’t quite handle… things which reminded me of how people had been describing IFS. So I finally read up on it, and have been successfully applying it ever since.

In this post, I’ll try to describe and motivate IFS in terms which are less likely to give people in this audience the same kind of a “no, that’s nonsense” reaction as I initially had.

Epistemic status

This post is intended to give an argument for why something like the IFS model could be true and a thing that works. It’s not really an argument that IFS is correct. My reason for thinking in terms of IFS is simply that I was initially super-skeptical of it (more on the reasons of my skepticism later), but then started encountering things which it turned out IFS predicted - and I only found out about IFS predicting those things after I familiarized myself with it.

Additionally, I now feel that IFS gives me significantly more

gears

for understanding the behavior of both other people and myself, and it has been significantly transformative in addressing my own emotional issues. Several other people who I know report it having been similarly powerful for them. On the other hand, aside for a few isolated papers with titles like “

proof-of-concept

” or “

pilot study

”, there seems to be conspicuously little peer-...
Future directions for narrow value learning
by Rohin Shah — 2019-01-26T02:36:51.532Z

Narrow value learning is a huge field that people are already working on (though not by that name) and I can’t possibly do it justice. This post is primarily a list of things that I think are important and interesting, rather than an exhaustive list of directions to pursue. (In contrast, the corresponding post for ambitious value learning did aim to be exhaustive, and I don’t think I missed much work there.)

You might think that since so many people are already working on narrow value learning, we should focus on more neglected areas of AI safety. However, I still think it’s worth working on because long-term safety suggests a particular subset of problems to focus on; that subset seems quite neglected.

For example, a lot of work is about how to improve current algorithms in a particular domain, and the solutions encode domain knowledge to succeed. This seems not very relevant for long-term concerns. Some work assumes that a handcoded featurization is given (so that the true reward is linear in the features); this is not an assumption we could make for more powerful AI systems.

I will speculate a bit on the neglectedness and feasibility of each of these areas, since for many of them there isn’t a person or research group who would champion them whom I could defer to about the arguments for success.

The big picture

This category of research is about how you could take narrow value learning algorithms and use them to create an aligned AI system. Typically, I expect this to work by having the narrow value learning enable some form of corrigibility.

As far as I can tell, nobody outside of the AI safety community works on this problem. While it is far too early to stake a confident position one way or the other, I am slightly less optimistic about this avenue of approach than one in which we create a system that is directly trained to be corrigible.

Avoiding problems with goal-directedness. How do we put together narrow value learning techniques in a way that doesn’t lead to the AI behaving like a goal-directed agent at each point? This is the problem with keeping a reward estimate that is updated over time. While reward uncertainty can help avoid some of the problems, it does not seem sufficient by itself. Are there other ideas that can help?

Dealing with the difficulty of “human values”. Cooperative IRL makes the unrealistic assumption that the human knows her reward f...
For what do we need Superintelligent AI?
by avturchin — 2019-01-25T15:01:01.772Z

Most of human problems could be solved by humans or slightly above human AIs. Which practically useful tasks – which we could define now – require very high level of superintelligence? I see two main such tasks:

1) Prevention of the creation of other potentially dangerous superintelligences.

2) Solving the task of indefinite life extension, which include solving aging and uploading.

I could imagine other tasks, like near-light-speed space travel, but they are neither urgent nor necessary.

For which other tasks do we need superintelligence?
Thoughts on reward engineering

by paulfchristiano — 2019-01-24T20:15:05.251Z

Note: This is the first post from part five: possible approaches of the sequence on iterated amplification. The fifth section of the sequence breaks down some of these problems further and describes some possible approaches.

Suppose that I would like to train an RL agent to help me get what I want.

If my preferences could be represented by an easily-evaluated utility function, then I could just use my utility function as the agent’s reward function. But in the real world that’s not what human preferences look like.

So if we actually want to turn our preferences into a reward function suitable for training an RL agent, we have to do some work.

This post is about the straightforward parts of reward engineering. I’m going to deliberately ignore what seem to me to be the hardest parts of the problem. Getting the straightforward parts out of the way seems useful for talking more clearly about the hard parts (and you never know what questions may turn out to be surprisingly subtle).

The setting

To simplify things even further, for now I’ll focus on the special case where our agent is taking a single action a. All of the difficulties that arise in the single-shot case also arise in the sequential case, but the sequential case also has its own set of additional complications that deserve their own post.

Throughout the post I will imagine myself in the position of an “overseer” who is trying to specify a reward function R(a) for an agent. You can imagine the overseer as the user themselves, or (more realistically) as a team of engineer and/or researchers who are implementing a reward function intended to expresses the user’s preferences.

I’ll often talk about the overseer computing R(a) themselves. This is at odds with the usual situation in RL, where the overseer implements a very fast function for computing R(a) in general (“1 for a win, 0 for a draw, -1 for a loss”). Computing R(a) for a particular action a is strictly easier than producing a fast general implementation, so in some sense this is just another simplification. I talk about why it might not be a crazy simplification in section 6.

Contents

Long time horizons. How do we train RL agents when we care about the long-term effects of their actions?
Inconsistency and unreliability. How do we handle the fact that we have only imperfect access to our preferences, and different querying strategies are not guaranteed to yie

...