trentmkelly/LessWrong-43k huggingface dataset:
Free Speech as Legal Right vs. Ethical Value
by ozymandias — 2017-11-28T16:49:38.540Z

[Crossposted from my blog with minor edits.]

It is very annoying that the English language does not distinguish between free speech the ethical value and free speech the legal right. I feel like many conversations would be far clearer and more sensible if different words were used for these two things.

For one thing, while most people who value free speech as a matter of ethics support the legal right to free speech, the converse is not true. I can believe deep in my heart of hearts that some speech ought to be censored-- that the people who speak it ought to be punished in various ways up to and including being punched in their stupid faces-- without thinking that it would be a good idea to give the US government the power to censor that speech.

The reason is that I cannot give the US government the narrowly tailored Power To Censor Only Speech Ozy Doesn't Like. I can only give them the power to censor speech that is generally unpopular. The one thing everything currently censored by the US government has in common is that it is wildly unpopular, whether for good reasons (saying false and malicious things about people, shouting at an angry torch-wielding mob that God requires you to burn witches) or bad ones (put your favorite example here).

Of course, much speech I do think should happen is very unpopular and much speech I think shouldn't happen is very popular.

So legal free speech is in the interest of all groups who want to say unpopular things. You get weird fucking allies in free speech defense: for example, a recent ACLU case's defendants included PETA (message: "go vegan"), an abortion provider (message: "abortion pills available cheaply, quickly and privately"), the ACLU itself (message: the text of the First Amendment), and Milo Yiannopoulous (message: "everyone hates me! buy my book"). These groups have nothing in common with each other except that lots of people would like them to shut up and go away. But a legal precedent that protects any one of them would protect all four.

I am always puzzled that centrists are generally more fervent defenders of the legal right to freedom of speech than leftists are. It seems to me there ought to be a whole bunch of centrists going "wait, you mean we can silence the alt-right, the Nazis, the communists, Black Lives Matter, and PETA all at once?"

Some people arg...
Stable agent, subagent-unstable
by Stuart_Armstrong — 2017-11-28T16:05:53.000Z

A putative new idea for AI control; index here.

A reflectively consistent agent is one that is ok with creating copies with the same motivation as itself, or continuing to have the same motivation under self-modification.

A reflectively stable agent is one that would only create agents with the same motivation as itself (unless rewarded for doing otherwise), and would actively want to preserve its motivation under self-modification.

Here is a design that is reflectively stable for its own self-modification, but not even reflectively consistent for copies and subagents.

Subtracting the value function

Let ht be the history of the agent's interaction with the environment up to time t.

Let the agent A have policy πA. Let V(R,π,ht) be the expected value of (future) reward R under policy π, given history ht. Let πR the policy that maximises this value the most.

Initially, the agent has no reward. Then, at time t, the AI gets reward function R, and gets the penalty:

−cV(R,πA,ht).

For c<1, this change of reward is reflexively stable (the agent will choose πA=πR to get maximal expected reward (1−c)V(R,πR,ht)), for c=1, it is reflexively consistent (any policy πA gives the same expected reward of 0), and for c>1, it is neither.

But instead, imagine the agent gets the penalty:

−cV(R,πR,ht).

In this case, the agent is reflexively stable for all values of c! That's because the penalty term is independent of its policy choice, and it will therefore choose πA=πR after t to maximise the reward it does get from R.

No subagent stability.

Using πR in the reward may make the agent itself reflexively stable, but if c>1, it will not copy the same goal into its subagent.

This is easy to see, because the agent's expected reward is (1−c)V(R,πR,ht), so its best policy is to minimise V(R,πR,ht). It will do this by acting so that ht is one the minimises that values.

For instance, it might create subagents whose task is to actively restrain the agent after t. Then it will struggle against them after t (because it now wants to maximise the reward R), but will set them up to be as powerful as possible before that, because before t, it wants its own future struggles to fail.

Clearly, though this is technically reflectively stable, this is not a good sort of stability.

Not 'real' stability

It should be noted that the agent's "stability" is an artefact of the fact that πR is defined

...
Stable agent, subagent-unstable
by Stuart_Armstrong — 2017-11-28T16:04:02.612Z

Crossposted at the intelligent agent forum.

A reflectively consistent agent is one that is ok with creating copies with the same motivation as itself, or continuing to have the same motivation under self-modification.

A reflectively stable agent is one that would only create agents with the same motivation as itself (unless rewarded for doing otherwise), and would actively want to preserve its motivation under self-modification.

Here is a design that is reflectively stable for its own self-modification, but not even reflectively consistent for copies and subagents.

Subtracting the value function

Let ht be the history of the agent's interaction with the environment up to time t.

Let the agent A have policy πA. Let V(R,π,ht) be the expected value of (future) reward R under policy π, given history ht. Let πR be the policy that maximises this value the most.

Initially, the agent has no reward. Then, at time t, the AI gets reward function R, and gets the penalty:

−cV(R,πA,ht).

For c<1, this change of reward is reflexively stable (the agent will choose πA=πR to get maximal expected reward (1−c)V(R,πR,ht)), for c=1, it is reflexively consistent (any policy πA gives the same expected reward of 0), and for c>1, it is neither.

But instead, imagine the agent gets the penalty:

−cV(R,πR,ht).

In this case, the agent is reflexively stable for all values of c! That's because the penalty term is independent of its policy choice, and it will therefore choose πA=πR after t to maximise the reward it does get from R.

No subagent stability.

Using πR in the reward may make the agent itself reflexively stable, but if c>1, it will not copy the same goal into its subagent.

This is easy to see, because the agent's expected reward is (1−c)V(R,πR,ht), so its best policy is to minimise V(R,πR,ht). It will do this by acting so that ht is one the minimises that value.

For instance, it might create subagents whose task is to actively restrain the agent after t. Then it will struggle against them after t (because it now wants to maximise the reward R), but will set them up to be as powerful as possible before that, because before t, it wants its own future struggles to fail.

Clearly, though this is technically reflectively stable, this is not a good sort of stability.

Not 'real' stability

It should be noted that the agent's "stability" is an artefact ... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Reward learning summary
by Stuart_Armstrong — 2017-11-28T15:55:08.000Z

A putative new idea for AI control; index here.

I've been posting a lot on value/reward learning recently, and, as usual, the process of posting (and some feedback) means that those posts are partially superseded already - and some of them are overly complex.

So here I'll try and briefly summarise my current insights, with links to the other posts if appropriate (a link will cover all the points noted since the previous link):

I'm modelling humans as a pair (p,R), where R is the reward function, and p is a planning algorithm (called a planer) that maps rewards to policies. An agent is trying to learn R.

The policy of a given human is designated πH. A pair (p,R) is compatible if p(R)=πH.

There is a no-free-lunch theorem for these (p,R) pairs. Once the agent has learnt πH, it can get no further evidence from observing the human. At that point, any compatible pair (p,R) is a valid candidate for explaining the human planner/reward.

Thus the agent cannot get any idea about the reward without making assumptions about the human planner (ie the human irrationality) - and can't get any idea about the planner without making assumptions about the human reward.

Unlike most no-free-lunch theorems, a simplicity prior does not remove the result. LINK

It's even worse than that: a simplicity prior can push us away from any "reasonable" (p,R). LINK

Ignoring "noise" doesn't improve the situation: the real problem is bias. LINK

The (p,R) formalism can also model situations like the agent "overriding" the human's reward. LINK

There are Pascal's mugging type risks in modelling reward override, where a very unlikely (p,R) pair may still be chosen because the expected reward for that choice is huge. LINK

There is also the risk of an agent transforming human into rational maximisers of the reward its computed so far LINK.

Humans are not just creatures with policies, but we are creatures with opinions and narratives about our own values emotions, and rationality. Using these "normative assumptions", we can start converging on better (p,R) pairs. LINK LINK

Even given that, our values will remain underdefined, changeable, and open to manipulation. LINK

Resolving those problems with our values is a process much more akin to defining values that discovering them. LINK

There are important part of human values that are not easily captured in the (p,R) formalism. LINK
Big Advance in Infinite Ethics
by bwest — 2017-11-28T15:10:47.396Z

Summary

It is possible that our universe is infinite in both time and space. We might therefore reasonably consider the following question: given some sequences u=(u1,u2,…) and u′=(u′1,u′2,…) (where each ut represents the welfare of persons living at time t), how can we tell if u is morally preferable to u′?

It has been demonstrated that there is no “reasonable” ethical algorithm which can compare any two such sequences. Therefore, we want to look for subsets of sequences which can be compared, and (perhaps retro-justified) arguments for why these subsets are the only ones which practically matter.

Adam Jonsson has published a preprint of what seems to me to be the first legitimate such ethical system. He considers the following: suppose at any time t we are choosing between a finite set of options. We have an infinite number of times in which we make a choice (giving us an infinite sequence), but at each time step we have only finitely many choices. (Formally, he considers Markov Decision Processes.) He has shown that an ethical algorithm he calls “limit-discounted utilitarianism” (LDU) can compare any two such sequences, and moreover the outcome of LDU agrees with our ethical intuitions.

This is the first time that (to my knowledge), we have some justification for thinking that a certain algorithm is all we will "practically" need when comparing infinite utility streams.

Limit-discounted Utilitarianism (LDU)

Given u=(u1,u2,…) and u′=(u′1,u′2,…) it seems reasonable to say u≥u′ if

∑∞t=0(ut−u′t)≥0

Of course, the problem is that this series may not converge and then it’s unclear which sequence is preferable. A classic example is the choice between (0,1,0,1,…) and (1,0,1,0,…). (See the example below.)

LDU handles this by using Abel summation. Here is a rough explanation of how that works.

Intuitively, we might consider adding a discount factor 0<δ<1 like this:

∑t=0∞δt(ut−u′t)

This modified series may converge even though the original one doesn’t. Of course, this convergence is at the cost of us caring more about people who are born earlier, which might not endear us to our children.

Therefore, we can take the limit case:

liminfδ→1−∑∞t=0δt(ut−u′t)

This modified summand is what’s used for LDU.

LDU has a number of desirable properties, which are summarized on page 7 of this paper by Jonsson and Voorneveld. I won’t go into them much here other than to say th... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
USA v Progressive 1979 excerpt
by RyanCarey — 2017-11-27T17:32:10.425Z

In 1979, an interesting judgment was made regarding the publication of an alleged nuclear infohazard. Here is an excerpt from that preliminary injunction ruling, which was authored by Robert W. Warren, then a Wisconsin Eastern District judge.

---

"The Secretary of State states that publication will increase thermonuclear proliferation and that this would "irreparably impair the national security of the United States." The Secretary of Defense says that dissemination of the Morland paper will mean a substantial increase in the risk of thermonuclear proliferation and lead to use or threats that would "adversely affect the national security of the United States."

Howard Morland asserts that "if the information in my article were not in the public domain, it should be put there . . . so that ordinary citizens may have informed opinions about nuclear weapons."

Erwin Knoll, the editor of The Progressive, states he is "totally convinced that publication of the article will be of substantial benefit to the United States because it will demonstrate that this country's security does not lie in an oppressive and ineffective system of secrecy and classification but in open, honest, and informed public debate about issues which the people must decide."

The Court is faced with the difficult task of weighing and resolving these divergent views.

A mistake in ruling against The Progressive will seriously infringe cherished First Amendment rights. If a preliminary injunction is issued, it will constitute the first instance of prior restraint against a publication in this fashion in the history of this country, to this Court's knowledge. Such notoriety is not to be sought. It will curtail defendants' First Amendment rights in a drastic and substantial fashion. It will infringe upon our right to know and to be informed as well.

A mistake in ruling against the United States could pave the way for thermonuclear annihilation for us all. In that event, our right to life is extinguished and the right to publish becomes moot.

In the Near case, the Supreme Court recognized that publication of troop movements in time of war would threaten national security and could therefore be restrained. Times have changed significantly since 1931 when Near was decided. Now war by foot soldiers has been replaced in large part by war by machines and...
You Have the Right to Think
by Zvi — 2017-11-27T02:10:00.348Z

Epistemic Status: Public service announcement. We will then return to regularly scheduled programming.

Written partly as a response to (Robin Hanson): Why be Contrarian, responding to the book Inadequate Equilibria by Eliezer Yudkowsky

Warning: Applause lights incoming. I’m aware. Sorry. Seemed necessary.

We the people, in order to accomplish something, do declare:

You have the right to think.

You have the right to disagree with people where your model of the world disagrees.

You have the right to disagree with the majority of humanity, or the majority of smart people who have seriously looked in the direction of the problem.

You have the right to disagree with ‘expert opinion’.

You have the right to decide which experts are probably right when they disagree.

You have the right to disagree with ‘experts’ even when they agree.

You have the right to disagree with real experts that all agree, given sufficient evidence.

You have the right to disagree with real honest, hardworking, doing-the-best-they-can experts that all agree, even if they wouldn’t listen to you, because it’s not about whether they’re messing up.

You have the right to have an opinion even if doing a lot of other work would likely change that opinion in an unknown direction.

You have the right to have an opinion even if the task ‘find the real experts and get their opinions’ would likely change that opinion.

You have the right to update your beliefs based on your observations.

You have the right to update your beliefs based on your analysis of the object level.

You have the right to update your beliefs based on your analysis of object-level arguments and analysis.

You have the right to update your beliefs based on non-object level reasoning, on any meta level.

You have the right to disagree with parts of systems smarter than you, that you could not duplicate.

You have the right to use and update on your own meta-rationality.

You have the right to believe that your meta-rationality is superior to most others’ meta-rationality.

You have the right to use as sufficient justification of that belief that you know what meta-rationality is and have asked whether yours is superior.

You have the right to believe the object level, or your analysis thereof, if you put in the work, without superior meta-rationality.

You have the right to believe that someone else ha...
Security Mindset and the Logistic Success Curve
by Eliezer Yudkowsky — 2017-11-26T15:58:23.127Z

Follow-up to: Security Mindset and Ordinary Paranoia

(Two days later, Amber returns with another question.)

amber: Uh, say, Coral. How important is security mindset when you're building a whole new kind of system—say, one subject to potentially adverse optimization pressures, where you want it to have some sort of robustness property?

coral: How novel is the system?

amber: Very novel.

coral: Novel enough that you'd have to invent your own new best practices instead of looking them up?

amber: Right.

coral: That's serious business. If you're building a very simple Internet-connected system, maybe a smart ordinary paranoid could look up how we usually guard against adversaries, use as much off-the-shelf software as possible that was checked over by real security professionals, and not do too horribly. But if you're doing something qualitatively new and complicated that has to be robust against adverse optimization, well... mostly I'd think you were operating in almost impossibly dangerous territory, and I'd advise you to figure out what to do after your first try failed. But if you wanted to actually succeed, ordinary paranoia absolutely would not do it.

amber: In other words, projects to build novel mission-critical systems ought to have advisors with the full security mindset, so that the advisor can say what the system builders really need to do to ensure security.

coral: (laughs sadly) No.

amber: No?

coral: Let's say for the sake of concreteness that you want to build a new kind of secure operating system. That is not the sort of thing you can do by attaching one advisor with security mindset, who has limited political capital to use to try to argue people into doing things. “Building a house when you're only allowed to touch the bricks using tweezers” comes to mind as a metaphor. You're going to need experienced security professionals working full-time with high authority. Three of them, one of whom is a cofounder. Although even then, we might still be operating in the territory of Paul Graham's Design Paradox.

amber: Design Paradox? What's that?

coral: Paul Graham's Design Paradox is that people who have good taste in UIs can tell when other people are designing

...
An Intuitive Explanation of Inferential Distance
by RichardJActon — 2017-11-26T14:13:36.512Z

Let us represent our beliefs as nodes in a network. Most nodes in your network of beliefs have dependencies, they are connected to other beliefs the truth value of which they are contingent on.

Picture a subset of your network of beliefs, let us take a simplified section that looks like a tree, stripping away the complex interconnections to aid in visualization.

Beliefs B and C are predicated on belief A, beliefs D and E on belief B, and beliefs F and G on belief C. Thus belief A is need for beliefs B and C, belief B for D and E, and C for F and G. (see the figure)

The One: Apples grow on trees - lets us call this belief F.

The Other: What are trees? and why are you labeling your beliefs with letters?

The one is trying to convince the other of F but the other lacks belief C (the belief in trees), the one is unable to successfully graft the branch of their tree containing F onto the other's tree as their is no suitable node at which to affix it.

The one must then traverse back down their tree to node C and first convince the other of C. The one made the mistake of thinking the inferential distance between themself and the other was but 1 node when in actuality it was 2. The one must convince the other not merely that apples grow on trees but must also introduce the concept of trees.

The One: Trees are large upright plants with stiff bodies due in part to large amounts of lignin in some of their cell walls, a property not shared by they bendier cousins. There is one over their with apples growing on it. (points to a nearby apple tree)

Assessing the inferential distance between yourself and your interlocutor can often be accomplished by the asking of questions which prompt an explication of their belief structure. It also helps here if you have exercised good epistemic hygiene and can clearly identify the path you must take in order to converge. The concept of inferential distance is interestingly related to finding the double crux of a disagreement, you must traverse back through your belief network until you find the relevant point of difference in your network or the genuine source of your disagreement. Indeed inferential distance may be measured by the number of iterations of the double crux game algorithm you have to execute find the crux. Once the extent of the gulf between you has been assayed you may begin to devise a path to traverse it.

There is a distinction...