trentmkelly/LessWrong-43k huggingface dataset:
Our values are underdefined, changeable, and manipulable
by Stuart_Armstrong — 2017-11-06T10:59:38.000Z

A putative new idea for AI control; index here.

When asked whether "communist" journalists could report freely from the USA, only 36% of 1950 Americans agreed. A follow up question about Amerian journalists reporting freely from the USSR got 66% agreement. When the order of the questions was reversed, 90% were in favour of American journalists - and an astounding 73% in favour of the communist ones.

There are many examples of survey responses depending on question order, or subtle issues of phrasing.

So there are people whose answers depended on question order. What then are the "true" values of these individuals?

Underdetermined values

I think the best way of characterising their values is to call them "underdetermined". There were/are presumably some people for which universal freedom of the press or strict national security were firm and established values. But for most, there were presumably some soft versions of freedom of the press and nationalism, and the first question triggered one narrative more strongly than the other. What then, are their "real" values? That's the wrong question - akin to asking if Argentina really won the 1986 world cup.

Politicians can change the opinions of a large sector of the voting public with a single pronouncement - were the people's real opinions the ones before, or the ones after? Again, this seems to be the wrong question. But don't people fret about this inconsistency? I'd wager that they aren't really aware of this, because people are the most changeable on issues they've given the least thought to.

And rationalists and EAs are not immune to this - we presumably don't shift much on what we identify as our core values, but on less important values, we're probably as changeable as anyone. But such contingent values can become very strong if attacked, thus becoming a core part of our identity - even if it's very plausible we could have held the opposite position in a world slightly different.

Frameworks and moral updating

People often rely on a small number of moral frameworks and principles to guide them. When a new moral issue arises, we generally try and fit it into a moral framework - and when there are multiple ones that could fit, we can go in multiple directions, driven by mood, bias, tribalism, and many other contingent factors.

The moral frameworks themselves can and do shift, due to issues like tribalism, cognitive

...
Learning values, or defining them?
by Stuart_Armstrong — 2017-11-06T10:59:28.000Z

A putative new idea for AI control; index here.

Is there a teapot currently somewhere between the orbit of Earth and Mars? And who won the 1986 World Cup football match between England and Argentina? And what do these questions have to do with learning human values?

Underlying true reality?

Both those top questions are uncertain. We haven't scanned the solar system with enough precision to find any such teapot. And though the referee allowed the first Argentine goal, making it official, and though FIFA had Agentina progress to the semi-finales (they would eventually win the tournament) while England was eliminate... that goal, the "Hand of God" goal, was scored by Maradona with his hand, a totally illegal move.

In a sense, neither question can ever be fully resolved. Even if we fully sweep the the solar system for teapots in a century's time, it's still possible there might have been one now, and it then crashed into the sun, stopping us from ever finding it. And in the spirit of ambijectivity, the question of Argentina's victory (or whether it was a "proper" victory, or a "fair" victory) depends on which aspect of the definition of victory you choose to emphasise - the referee's call and the official verdict, versus the clear violation of the rules.

Nevertheless, there is a sense in which we feel the first question has a definite answer (which is almost certainly "no"), while the second is purely about definitions.

Answering the question

Why do we feel that the teapot question has a definite answer? Well, we have a model of reality as something that actually objectively exists, and our investigation of the world backs it up - when confronted by a door, we feel that there is something behind the door, even if we choose not to open it. There are various counterfactuals in which we could have sent out a probe to any given area of the orbit, so we feel we could have resolved the "is there a teapot at this location" for any given location within a wide radius of the Earth.

Basically, the universe has features that causes us to believe that when we observe it (quantum effects aside), we are seeing a previous reality rather than creating a new one (see the old debate between platonists and formalists in mathematics).

Whereas even if we had a God or an AI, we don't expect it to have a definite answer to the question of who won that football match. There is no platonic u

...
Self-confidence as a time saving tactic
by devas — 2017-11-06T10:37:56.940Z

Epistemic status: anecdotal evidence which I believe can be generalized.

–

An insight I had in the last few months is that self-confidence has a positive side-effect above and beyond the positive qualities usually associated with it. What I mean is: we've all heard of “just believe in yourself” is the panacea to nearly everything: this is about something different.

Self-confidence is an invaluable way of doing things quickly and efficiently, saving mental resources.

To illustrate my point, I’ll use a real life example: when encountering a problem during work (or study, or starting a relationship, or something else) a person with low confidence will usually agonize over what to do, seek outside advice, validation or orders, in an effort to take the “safe” path.

This has consequences; speaking personally, I had a horrible experience in a past job partly because of this; every question was met with scorn because I didn’t already know what I was supposed to do, or because “it’s obvious! Why are you even wasting my time?!?”.

This made me more insecure, thus leading to me stalling more when encountering something outside of my immediate area of expertise, agonizing over whether to ask questions or not, and so on and so forth.

As an aside, even though asking question is usually said to be a virtuous thing by everyone, this is not so in reality: there are stupid questions, and more to the point, there are questions that if voiced, will hurt your social standing and make you look stupid. Questions that, even if answered, aren't really worth the cost. (No one will volunteer to tell you what those question are. You have to figure it out yourself).

The crucial point of this is that by not asking questions/for orders and using their own judgment, a person is taking a risk (sometimes a very big risk, sometimes just what feels like one).

Except that sometimes taking a big risk is the correct thing to do.

Yes, even if the risk does not pay out and your boss/teacher/significant other berates you for half an hour and makes you feel like the lowest of worms.

There are obvious costs to this: taking risks, is, well, risky. The crucial decision to make is: is the time and reputation preserved by doing this worth the probability of failure?

A way of thinking about this is as an approximation: you treat the fearful/unknown situation as if it were something you already know how to d...
Acknowledging Rationalist Angst
by Hazard — 2017-11-06T05:26:45.505Z

Not all advice can work for everyone, and not all techniques are universally applicable. That being said, here’s a post that would have been incredibly useful for me-three-years-ago to have read, and hopeful it strikes home for some people who read it. I’ll start with drawing your attention to a character archetype that you’ve probably seen before.

*******************************

Blake lives in a textbook example of a dystopian totalitarian society. There is a cheery facade of being just and righteous, but secretly The Party does Really Bad Things. Blake has only ever known the world of The Party, and has always taken The Orthodoxy to be pretty much true. Well, sometimes it’s given him some weird feelings that he can’t quite pin down, but surely whatever faults the party may have, at least he doesn’t have to live in the anarchy of The Outside World. You know the story from here. Through a series of unlikely events, the lies of the party are revealed to him and he has to flee for his life. Blake’s entire understanding of the world has been shattered, and he’s alone, scared, and confused. Luckily, he’s picked up by a group of rebels, and is amazed to learn that there is a strong but small Underground Resistance. By now, Blake has had enough time for his fear and confusion to turn into feelings of betrayal and anger.

*********************************

I’m willing to go out on a limb and say that the core feelings that Blake was experiencing are the same core feelings I experienced in the beginning of my journey into rationality, and I’d guess that there is a non-negligible subset of people in this community who might have experienced something similar.

Okay, yes, the example is a little bit grandiose, but let’s look at the core. (I’ll be talking in terms of my specific journey, and hopefully there’s some crossover for other people’s lives) For me, reading the sequences and “becoming a rationalist” was the tipping point of a slowly forming realization that there was a lot more to thinking/decision making/reality/everything then what I was being taught by those around me. There were definitely feelings of resentment towards to the general society and system that I was raised by. There was a feeling of thinking that most of the world was insane and wasn’t aware of its insanity. There were feelings of confusion as I tried to piece together what I believed and why I beli...
Paradox and pedagogy
by whales — 2017-11-05T21:01:15.738Z

When water freezes, it expands. Thus, my latest invention places a weight on a column of water, then freezes the water to lift the weight, doing work. By using a weight heavy enough that the work done is greater than the energy expenditure to freeze the water, I create a perpetual motion machine.

The Second Law of Thermodynamics tells you that this is impossible. But it tells you so in a non-constructive way. It doesn't say which step is mistaken, what unmentioned physical mechanism is required for its satisfaction, or exactly how it works out mathematically. Even partial explanations—say, water under pressure takes more energy to freeze—aren't completely satisfactory, leaving questions like: Why does it work out no matter how much weight I load it with? How would it be different for something like mercury that contracts when freezing? The Second Law only tells you it's hopeless.

That's fine when you're deciding whether or not to invest in my startup, but if you're trying to learn about the world, you should feel a persistent itch, a gap where an explanation should go. And it's not the simple sensation of not knowing. You might describe it as tension, confusion, or even paradox.

There are (at least) two major reasons to pay attention to the use of paradox in teaching and learning. The first is that itch—it's motivating. There's not merely a blank space in your picture of the world—there's an apparent contradiction. If you try to fill in the gap from different directions, you get different answers. One of my major qualitative takeaways from my old self-experiment on noticing confusion was that in cases like this I didn't even need any kind of trigger-action plan to follow up on the noticing, since the need to resolve it was so powerful—and even having written that, I continue to underestimate the power of this effect.

So if you're psychologically anything like me, for the sake of motivation alone, I suggest you look for ways to manufacture contradictions out of gaps in your knowledge.

The second thing to pay attention to is what happens when you resolve the paradox. On the most benign level, you get a fuller picture of why things are the way they are when you see that they couldn't be some other way. In a sense, no explanation is complete without this. Going a bit deeper, doing this helps you develop...
Pascal's mugging in reward learning
by Stuart_Armstrong — 2017-11-05T19:44:59.556Z

This is the last post on the current value/reward learning series. I'll just bring up the danger of Pascal's mugging in reward learning.

I've already shown how you can use the (p, R) model (with R a reward and p a planning algorithm, called a planner) to model AI actions &quot;overriding&quot; human preferences. That post used heroin as an override example, but here we'll use an even more extreme example; the AI will do brain surgery to deliberately re-wire the human into following the policy of its choice:

The event S corresponds to the AI doing the surgery, the event ¬S to it not doing so. After this, the human will either follow the subsequent policy π(STA), a "standard" human policy according to some criteria, and π(MAX), another policy (why it is so named will soon become clear).

Let R(STA) and R(MAX) be rewards corresponding to π(STA) and π(MAX), under the assumption that the human policy is maximising them perfectly.

For any reward R, the value of that reward, V*(R), is the expected value of that reward if the AI acts perfectly to maximise it.

Now, the policy π(MAX) is the rational behaviour for maximising at least one reward R(MAX) - and, conversely, given R(MAX), there is a human policy π(MAX) that best maximises it. This allows us to the reward (and hence the policy) as:

R(MAX) = argmax(R) [V(R)|S], the reward with the most value conditional on S. Define Ω = max(R) [V(R)|S] = V*(R(MAX))|S.

Extend R(MAX) to the ¬S branch by defining R(MAX)' = I(¬S)R(STA)+I(S)R(MAX), with I(S) the indicator function for S.

Consider two planners: pr, the rational planner, and pf, the planner that is rational conditional on ¬S, and maps everything to π(MAX), conditional on S.

There are three compatible planner-reward pairs: (pr, R(MAX)'), (pf, R(STA)), and (pf, R(MAX)'). The first option corresponds to the human being fully rational, including after brain surgery from the AI, while the second encodes the fact that the brain surgery has overridden the human preference. The third pair also has an override, but, by pure coincidence, the override maps to the ideal policy the human wanted anyway.

Now, under any set of reasonable normative assumptions, (pf, R(STA)) is the correct pair. But, as good Bayesians, we don't want to set probabilities to zero, so assume

P(pr, R(MAX)') + P(pf, R(MAX)') = ε.

Thus the probability ...
Zeroing Out
by Zvi — 2017-11-05T16:10:00.760Z

Related to (Eliezer Yudkowski at Less Wrong): An Equilibrium of No Free Energy

Related to (Satvik Beri at Less Wrong): Competitive Truth Seeking

Follow-up to: Leaders of Men

I: Markets

Suppose you have an insight about Google. The efficient market hypothesis says you can’t make a profit. Your insight is not a new insight. The market has already priced it in. You know no more about Google’s future price than you did before.

That’s the bad news. That’s also the good news: If you didn’t have that insight , you wouldn’t know any less about Google’s future price. Efficient market!

I call this zeroing out. Your ignorance is not punished.

This means any unique knowledge you do find will be rewarded. If you get good Google news first, you don’t need to know anything else. Buy, buy, buy!

From Leaders of Men:

It is much, much easier to pick out a way in which a system is sub-optimal, than it is to implement or run that system at anything like its current level of optimization.

A corollary of this:

If is much, much easier to find a bias in a complex model, than it is to build an equally good model.

“I built a better model of Google’s future profits than the market” is a hard sell. “I found a factor the market isn’t properly accounting for” or “I got this news and figured out what it means before the market could price it in” is still a tough sell – it’s Google – but it’s less impossible.

Taking difficulty down, consider odds on the winner of a football game. Both tasks are now realistic. You would still have a much easier task figuring out “Alabama wins more often than people think” than “Alabama is 95.3% to win.”

The easiest way to beat consensus opinion is to use consensus opinion as an input.

It’s easy to beat the wisdom of the crowd a little – average your opinion into the crowd.

II Hiring

Hiring an engineer is trickier. Assume a consensus interview process.

Satvik suggests focusing on finding candidates like Bob, who does poorly on interviews but is an excellent employee, rather than candidates like Alex, who is an excellent employee and also an excellent interview. Alex likely won’t take the job, but Bob will. A...
Non-market failures: inefficient networks
by whales — 2017-11-05T04:10:21.376Z

Economics has a rich language for talking about market failures. Situations involving things like externalities, asymmetric information, public goods, or principal-agent problems can all result in room for improvement. In practice these situations haven't turned out to be fully tractable, but they're broadly recognized for what they are. We can at least talk theoretically about what's wrong and how to fix it in a market-failure framework, and sometimes we can use the question "how does this deviate from this idealized mechanism (which we believe brings about the best outcomes)?" to guide effective interventions. But not all failures are market failures—not even all failures concerning allocation of goods and services, broadly construed.

Within organizations, we may not expect or want allocation to be governed by price-market mechanisms. For example, firms should do things in-house when they have a strong internal hierarchy and transaction costs are high externally. "Organizational failure" can be a framework for thinking about when that goes wrong due to poor assumptions about the "ideality" of the organization.

And when a system necessarily involves non-hierarchical social (non-market) relationships, especially if there's a continuous search for information and a high need for trust, you want to think about what networks should be doing but aren't.

From The Anatomy of Network Failure (Schrank & Whitford 2011):

When we refer to network failures, we mean failure in a sense that self-consciously parallels what is meant in the literatures on market and organizational failure. In order to define the term, therefore, we begin with, but then tweak, a definition and an approach to definition that has its roots in a seminal 1958 paper (“The Anatomy of Market Failure”) by Francis Bator. “What is it we mean by ‘market failure’?” Bator (1958:351) asked. He answered: “Typically, at least in allocation theory, we mean the failure of a more or less idealized set of price-market institutions to sustain ‘desirable activities’ activities or to estop ‘undesirable’ activities.” His goal was to highlight that a market fails relative to some purpose—often but not necessarily the maximization of efficiency or welfare—and, simultaneously, to direct attention toward variation in the mechanisms driving those outcomes.

The authors...
Moved from Moloch's Toolbox: Discussion re style of latest Eliezer sequence
by habryka — 2017-11-05T02:22:48.471Z

Habryka: I considered moving this thread to meta, since I think it's highest value to separate the presentational conversation from the object-level conversation. I asked Hypothesis whether he would be fine with it, he said yes, and seemed to be more on board after I pointed out that Eliezer does in fact have an editor.

Comment by Hypothesis on Moloch's Toolbox:

Eliezer, this might come off as just another one of those five hundred screaming indignant people unreasonably asking for your assistance, but please listen to me.

Get an editor. Get a steely eyed, combative editor that will prevent you from putting sentences like this into your writing:

And if I named more than three supercategories, you wouldn’t be able to remember them due to computational limitations (which aren’t on the list anywhere, and I’m not going to add them).

Right now, this post is a beautiful esoteric summary of some of the most wrong things in the world, the forces and factors behind the civilizational Alzheimer&#x27;s which seems to progress continually. A disease that steamrolls fake tanned demagogues who are barely even capable of understanding the horror of what they&#x27;re up against. And with it you're preaching to the choir, the people who need to hear your message are not the ones you'll reach with these essays.

I do not have a baby that is dying because doctors can't buy the right formula. I am not on the committee of a central bank. Your post seems to be written to me but also tells me I am not in a position to fix the problems you outline. I detect a hint of the sentiment in John 15:19:

If you belonged to the world, the world would love you as one of its own. But because you do not belong to the world and I have chosen you out of it, the world hates you.

And that's strange to me. You previously wrote that rationality is systemized winning, but then shy away from acknowledging the world comes as it is and must be fought in some way on its terms.

If you think every institution is locked into stupid incentives and the only thing that sometimes gets stuff changed is public outrage, well I'm not the one that controls that. And your writing isn't targeted at the people who are, in fact I'm not sure who it's targeted at besides maybe someone who already understands the situation. The people who control this world are of the world ...