trentmkelly/LessWrong-43k huggingface dataset:
Relative exchange rate between preferences
by Stuart_Armstrong — 2019-03-29T11:46:35.285Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

In my initial post on synthesising human preferences, I imagined a linear combination of partial preferences. Later, when talking about identity preferences, I proposed a smoothmin instead (which can also be seen as strong diminishing returns to any one partial preference).

I was trying to formalise how humans seem to trade-off their various preferences in different circumstances. However the ideal is not to a priori decide how humans are trading off the preferences, but instead to copy how humans actually do trade off the preferences.

To do that, we need to imagine the human in situations quite distant from their current ones - situations where some of their partial preferences are more or less fulfilled. This brings up the problem of modelling preferences in distant situations. Assuming some acceptable resolution to that problem, the AI would have an exchange rate between different preferences, in different situations and with different levels of preference fulfilment.

Meta-preferences may constrain the preferences in very distant situations. Most paradoxes of population ethics are in situations very different from today, so population ethics preferences can be seen in this way. Universal moral principles act in a similar way, giving limits on what can happen in all situations, including extreme ones - though note there are arguments to avoid some unusual situations entirely.

Then the AI's task is to come up with a general formula for the exchange rate between different preferences, that extends to all situations and respects the constraints of the meta-preferences. It will probably smooth out quite a bit of "noise" in the exchange rates between different preferences, while respecting the general trends.
Being wrong in ethics
by Stuart_Armstrong — 2019-03-29T11:28:55.436Z

"I ache for the day when I am in heaven singing the praise of God".

That's a preference that many religious believers have. Not being religious myself, I don't have it, and even find it factually wrong - but how can a preference be wrong?

Here's a preference I do have:

"I ache for a world where people spontaneously help each other overcome their problems".

That preferences is also "wrong", in a similar sense. Let's see how.

Decomposing partially erroneous variables

For the purpose of this post, I am assuming atheism, which is the position I myself hold; for those of a religious disposition, it may help to assume that "God" refers here to a non-existent god from a different, wrong, religion.

Now, the "praising God" preference could be decomposed into other, simpler variables. It may feel like it's a single node, but it is likely composed of many different preferences linked together:

So, for example, those with that preference might think it good to be humble, feel there's something intrinsically valuable about holiness, enjoy the ecstatic feeling they get when praising God in a religious context, and feel that one should praise God out of a feeling of gratitude. We can consider that these form four of the foreground variables that the believer "cares about" in their partial model (what about the fifth node? We'll get back to that).

Note that the human didn't necessarily use a partial model with these four variables; they may instead have just a single variable entitled "praise god in heaven". However, we're decomposing the partially erroneous variable (since God doesn't exist), using the believer's web of connotation, into simpler variables.

Erroneous foreground variables

There's nothing wrong with the humbleness and ecstatic preferences. The gratitude node is problematic, because there ain't any God. If the believer only feels gratitude to God, then we can just erase that node as an error. However, it's possible that they might feel general gratitude towards their superiors, or towards their parents/mentors/those who've made them who they are. In that case, the preference for "Gratitude to God" is simply a special case of a very valid preference applied to an erroneous situation.

Then we have the interesting preference for "holiness". Holiness is not a well-grounded variable, but has its own web of connotation, maybe consisting of religious rituals, feelings

...
Models of preferences in distant situations
by Stuart_Armstrong — 2019-03-29T10:42:14.633Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

For X, consider three different partial preferences:

If X were poor, they would prioritise consumption over saving.

X: If I were poor, I would prioritise saving over consumption.

X: If I were poor, I'd get my personal accountant to advise me on the best saving/consumption plan for poor people.

1 is what X's judgement would be in a different, distant situation. 2 is what X's current judgement about what their judgement would be in that situation. 3 is similar, but is based on a factually wrong model of what that distant situation is.

So what are we to make of these in terms of X's preferences? 3 can be discounted as factually incorrect. 2 is a correct interpretation of X's current (meta-)preferences over that distant situation, but we know that these will change if they actually reach that situation. It might be tempting to see 1 as the genuine preference, but that's tricky. It's a preference that X doesn't have, and may never have. Even if X were certain to end up poor, their preference may depend on the path that they took to get there - medical bankruptcy, alcoholism, or one dubious investment, could result in different preferences. And that's without considering the different ways the AI could put X in that situation - we don't want the AI to influence its own learning process by indirectly determining the preferences it will maximise.

So, essentially, using 1 is a problem because the preference is many steps removed and can be influenced by the AI (though that last issue may have solutions). Using 2 is a problem because the current (meta-)preferences are projected into a situation where they would be wrong. This can end up with someone railing against the preferences of their past self, even if those preferences now constrain them. This is, in essence, a partial version of the Gödel-like problem mentioned her, where the human rebels against the preferences the AI has determined them to have.

So, what is the best way of figuring out X's "true" preferences? This is one of the things that we expect the system to be robust to. Whether type 1 or type 2 preferences are prioritised, the synthesis should still reach an acceptable outcome. And the rebellion against the synthesised values is a general problem with these methods, and

...
Please use real names, especially for Alignment Forum?
by Wei Dai — 2019-03-29T02:54:20.812Z

As the number of AI alignment researchers increases over time, it's getting hard for me to keep track of everyone's names. (I'm probably worse than average in this regard.) It seems the fact that some people don't use their real names as their LW/AF usernames makes it harder than it needs to be. So I wonder if we could officially encourage people to use their real firstname and lastname as their username, especially if they regularly participate on AF, unless they're deliberately trying to keep their physical identities secret? (Alternatively, at least put their real firstname and lastname in their user profile/description?)
Parfit's Escape (Filk)
by Gordon Seidoh Worley — 2019-03-29T02:31:42.981Z

To the tune of "Escape" (The Piña Colada Song), with apologies to Rupert Holmes.

I was lost in the desert, hopelessly dying of thirst
And I thought to myself, this can't get any worse
I heard the roar of an engine, surely it was my hearse
But then a tall shadow cooled me, it spoke and I heard

"If you like living not dying, I got an offer for you
If you give me your money, I'll give you water and food
I'll take you away to salvation, I'll get you out of this scrape
So just promise you'll pay me, to make Parfit's escape"

I hadn't solved decision theory, boy I sure wish that I had
'Cause my savior was Omega, and if I lied I'd be had
But CDT said don't pay, just take the ride for free
And though it seemed kind of foolish, I went ahead and agreed

"Yes, I like living not dying, so I'll make your offer good
When we get back to town, I've got money for you
I've got to get out of this desert now, I'm so tired of this place
Yes I two box on Newcomb, and I take Parfit's escape"

Omega dragged me to the car, put a canteen to my lips
We drove into the sunset, I felt nothing but bliss
I caught a glimpse of his face, and to my wondering eyes
My driver wasn't Omega, but Singer in disguise

"You don't owe me any money, you're the life I can save
And you should know I would have helped you, if you had lied to my face
I really hope you've learned a lesson though, about the perils of this place
Decision problems are dangerous, it's moral luck you escaped"

I said "Oh thank you Mr. Singer, let me buy you a drink
At a bar called O'Malley's, they make their Coladas pink
'Cause I'm eternally grateful, for all that you've done
Now let's get out of this desert, and be done with this song"
The trolley problem, and what you should do if I'm on the tracks
by Arthur Milchior — 2019-03-29T01:24:43.655Z
Link: http://www.milchior.fr/blog_en/index.php/post/2019/03/29/The-trolley-problem%2C-and-what-_you_-should-do-if-I-m-on-the-tracks

Originally published in French. Translation by Épiphanie.

Trigger warning: Death, suicide, and murder. Trolley problem.

This is quite the conventional and ethical conundrum: You are near train tracks, and a train is rolling down the hill. It is going to run over 4 people who are tied to the rails of the main track. However, you can change the train's direction to a secondary track by pulling a lever; so that it runs over only one guy, also tied down the rails. Should you pull the lever?

I do believe there is a more interesting way to frame it: What would you choose if you are yourself tied to the rails, alone, while the train is not heading toward you yet. My own answer is very simple: I want the person deciding where the train should go to have no doubts they should pull the lever! Because, for lack of context, I assume that the other four people are just me, or rather copies of me. That's a bit simplistic, of course they are not perfect clone. But as far as concrete predicates go, they are indistinguishable. That is to say I have odds of being on tracks alone of 1 in 5, and odds for being in the group of 4 in 5.
And tell you what, I prefer dying with 20% probability because of what someone did, rather than to die with 80% probability because no one was ever willing to take the burden of responsibility.

I know many would not pull the lever, or at least be very reluctant to. That is precisely the reason I am writing this post: I wish to make it public that I believe people should pull the lever. More importantly, I wish that many many more people would also share publicly this opinion. Then, if it is of public notoriety, the one who has to pull the lever would know they can do it without any remorse, as they will not have to face any societal consequences for what they've done. So all in all, this would raise my odds of survival of 60 points! That's quite something

But what if it were to truly happen?

Be aware that I am not saying anything more than what I have literally written. I have full rights over inconsistencies. Were I to be on the tracks alone, there is no telling what I would do. Maybe I would cry and plead not to pull the lever, maybe I would depict the other four people as worse than Hitler, or I would indulge the person into thinking they'd bear the burden of my death. Once I know that the probability of me dying jumps from 20 to 100%, I would -I pr

...
Announcing predictions
by Nathan Galt — 2019-03-28T21:01:17.102Z

Today I unveiled predictions, a command-line program to score predictions you’ve written down in a YAML file. In my estimation, the program and its supporting documentation is alpha quality. You’ll need to build it yourself and the documentation isn’t all there and well-organized yet.

If you’re familiar with Go toolchains, have a look. I’d be happy to take feedback here in addition to on GitHub.

https://github.com/adiabatic/predictions
What would you need to be motivated to answer "hard" LW questions?
by Raemon — 2019-03-28T20:07:48.747Z

Edit: Significantly rewritten. Original question was more specifically oriented around money-as-a-motivator.

One of the questions (ha) that we are asking ourselves on the LW team is "can the questions feature be bootstrapped into a scalable way of making intellectual progress on things that matter."

Motivations

Intrinsic vs Extrinsic

I'd cluster most knobs-to-turn here into "intrinsic motivation" and "extrinsic motivation."

Intrinsic motivation covers things like "the question is interesting, and specified in a way that is achievable, and fun to answer."

Extrinsic motivation can include things like "karma rewards, financial rewards, and other things that explicitly yield higher status for

(Things like "I feel a vague warm glow because I answered the question of someone I respect and they liked the answer" can blur the line between intrinsic and extrinsic motivation)

Improving Intrinsic Motivation

Right now I think there's room to improve the flow of answering questions:

New features such as the ability to spawn related questions that break down a confusing question into an easier.
Better practices/culture, such as as a clearer idea of how to specify questions such that they communicate what one needs to do to solve them (or, have a set of common practices among answerers such that this is easier to figure out).
A combination (wherein best practices are communicated via tooltips or some-such)

Bounties and Reliability

A lot of questions are just hard to answer – realistically, you need a lot of time, at least some of the time won't be intrinsically fun, and the warm glow of success won't add up to "a few days to a few months worth of work."

So we're thinking of adding some more official support for bounties. There have been some pretty successful bounty-driven content on LW (such as the AI Alignment Prize, the Weird Aliens Question, and Understanding Information Cascades), which have motivated more attention on questions.

Costly signaling of value

They showcase that the author of the question cares about the answer. Even if the money is still relatively minor, it reaffirms that if you work on the question, someone will actually derive value from it, which can be an actual important part of intrinsic motivation (as well as a somewhat legible-but-artificial status game you to more easil...
[Link] IDA: 11-14/14: Future Directions
by RAISE — 2019-03-28T18:56:47.784Z

This is a linkpost for https://app.grasple.com/#/level/1553

Every Thursday for 4 weeks, we've been posting lessons about Iterated Distillation and Amplification. They're largely based on Paul Christiano's sequence here on LW who has graciously allowed us to use his work.

This is the final section, explaining the next steps, the current holes in the plan and how they might be fixed.

Also look out for our continuing video sequence on IA which will be posted on Robert Miles' Youtube channel as well as on our course.

Note that access to the lessons requires creating an account here.

Have a nice day!
Alignment Newsletter #50
by Rohin Shah — 2019-03-28T18:10:01.264Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter.

Highlights

More realistic tales of doom

(Paul Christiano): This

Vox article

does a nice job of explaining the first part of this post, though I disagree with its characterization of the second part.

The typical example of AI catastrophe has a powerful and adversarial AI system surprising us with a treacherous turn allowing it to quickly take over the world (think of the paperclip maximizer). This post uses a premise of continuous AI development and broad AI deployment and depicts two other stories of AI catastrophe that Paul finds more realistic.

The first story is rooted in the fact that AI systems have a huge comparative advantage at optimizing for easily measured goals. We already see problems with humans optimizing for the easily measured goals (scientific malpractice, outrage-inducing social media, etc.) and with AI these problems will be severely exacerbated. So far, we have been able to use human reasoning to ameliorate these problems, by changing incentives, enacting laws, or using common sense to interpret goals correctly. We will initially be able to use human reasoning to create good proxies, but over time as AI systems become more capable our ability to do this will lag further and further behind. We end up "going out with a whimper": ultimately our values are no longer shaping society's trajectory.

The second story starts out like the first story, but adds in a new complication: the AI system could develop internal goals of its own. AI performs a huge search over policies for ones that score well on the training objective. Unfortunately, a policy that optimizes for the goal of "having influence" will initially score well on most training objectives: when you don't already have influence, a good strategy for gaining influence is to do what your overseers want you to do. (Here "influence" doesn't mean just social influence; control over nukes also counts as influence.) At some point the system will be powerful enough that gaining influence no longer means doing what the overseers want. We will probably know about this dynamic through some catastrophic AI failures (e.g. an AI-run corporation stealing the money it manages), but may not be able to...
Timing through the temporal dimension. 
by Furyan — 2019-03-28T13:21:31.468Z

A completely new way of thinking about time.

Our universe exists in a 2 dimensional matrix which we call space-time. It consists of a spatial dimension, interlinked with a temporal dimension. The word 'time' is a term describing temporal motion. So we 'move' through the spatial dimension and we 'time' through the temporal dimension.

Motion through the spatial dimension is measured in inches, meters, kilometers, light years, etc. Motion through the temporal dimension is measured in seconds, hours, days, years, etc. The speed at which we time through the temporal dimension is called our temporal velocity.

We only time in one direction. Towards the future. So the future is a temporal location, towards which we 'time'. The past is a temporal location which we have already occupied.

Because the two dimensions are interlinked, our spatial velocity is inversely proportional to our temporal velocity. The faster we move, the slower we time. Timing, however, is not just for humans. Everything in the universe, besides mass less particles, times.

To us it feels as though time is something that passes, but it's like seeing trees pass the window of a moving train. It's our temporal motion which causes the illusion of time passing. You can think of a clock as a temporal odometer. It measures how far we time. So tape measures measure spatial distances and clocks measure temporal distances.

Velocity is the relationship between moving and timing.

This is a work in progress, but feel free to comment.
What are CAIS' boldest near/medium-term predictions?
by Bird Concept — 2019-03-28T13:14:32.800Z

Background and questions

Since Eric Drexler publicly released his “Comprehensive AI services model” (CAIS) there has been a series of analyses on LW, from rohinmshah, ricraz, PeterMcCluskey, and others.

Much of this discussion focuses on the implications of this model for safety strategy and resource allocation. In this question I want to focus on the empirical part of the model.

What are the boldest predictions the CAIS model makes about what the world will look in <=10 years?

“Boldest” might be interpreted as those predictions which CAIS gives a decent chance, but which have the lowest probability under other “worldviews” such as the Bostrom/Yudkowsky paradigm.

A prediction which all these worldviews agree on, but which is nonetheless quite bold, is less interesting for present purposes (possibly something like that we will see faster progress than places like mainstream academia expect).

Some other related questions:

If you disagree with Drexler, but expect there to be empirical evidence within the next 1-10 years that would change your mind, what is it?
If you expect there to be events in that timeframe causing you to go “I told you so, the world sure doesn’t look like CAIS”, what are they?

Clarifications and suggestions

I should clarify that answers can be about things that would change your mind about whether CAIS is safer than other approaches (see e.g. the Wei_Dai comment linked below).

But I suggest avoiding discussion of cruxes which are more theoretical than empirical (e.g. how decomposable high-level tasks are) unless you have a neat operationalisation for making them empirical (e.g. whether there will be evidence of large economies-of-scope of the most profitable automation services).

Also, it might be really hard to get this down to a single prediction, so it might be useful to pose a cluster of predictions and different operationalisations, and/or using conditional predictions.
General v. Specific Planning
by james_t — 2019-03-27T12:34:07.762Z

Epistemic Status: Everyone already knows about it?

I've been thinking a bit about two different manners of pursuing a goal.

I haven't come up with a catchy jargon for them, and I don't know of any existing catchy jargon for them either. General v. specific planning is pretty bad, but at least for the purpose of this post I'll stick to it.

I know they've been discussed here in one form or other, probably many times, but I don't think they've really been explicitly contrasted. I thought doing that might be useful.

Here are some suggestive, if imperfect, contrasts, illustrating what I mean.

Examples

How I try to win Chess versus how Chess grandmasters and AlphaZero try to win Chess.

I tend to play Chess by trying to get an advantage in valuable pieces, because that is generally useful to me. I then try to use these pieces to eventually obtain checkmate.
On the other hand, AlphaZero seems to play to obtain a specific, although gradually accumulated, positional advantage that ultimately results in a resounding victory. It is happy to sacrifice "generally useful" material to get this.
This isn't simply a matter of just using different techniques to get to the end. It has more to do with my inability to identify strong positions and picture a game very far into the future.

Peter Thiel's "indefinite optimism" about career success versus "definite optimism" about career success.

According to this schema, the typical indefinite optimist's life-path consists in getting instrumentally useful things, such as education, status, or money, without committing to a definite course of action. The stereotypical career for such a person is finance or consulting or "business." Their success is supposed to follow the pursuit of optionality.
The definite optimist's life path, on the other hand, is more likely to consist in researching and shooting for a single, particular course of action. The stereotypical career for such a person is as an inventor or entrepreneur. Their success is supposed to follow after giving up a great deal of optionality.
The indefinite optimist accumulates generally useful resources, while the definite optimist will give up many "generally useful" things in order to reach a single goal.

OpenAI's strategy versus DeepMind's strategy of handling AGI s

...