trentmkelly/LessWrong-43k huggingface dataset:
Value learning for moral essentialists
by Charlie Steiner — 2019-05-06T09:05:45.727Z

Many people - people we want to persuade - are essentialists about morality and ethics. They give weight to the idea that just like how knowing facts about my shoes is possible because my shoes exist and interact with me, facts about what is right or good are possible because there is some essence of rightness or goodness "out there" that we somehow interact with.

This isn't totally wrong. But I think it reflects the usual human heuristic of assuming that every category has an essence that makes it what it is. Put a human race in a world with water, air, and fire, and the first thing they'll think is that the basic categories of water, air, and fire must somehow be reflected in the fundamental order of the world. And why does heat behave the way it does? The flow of heat substance. Why does opium make you drowsy? It has sleepiness essence.

This sort of heuristic passes for everyday use, but if you assume that goodness is a fundamental part of the world, the idea of an AI learning to do the right thing is going to sound a bit weird. How can we talk about value learning without even once checking if the AI interacts with the goodness-essence?

This post outlines the sort of strategy I'd use to try to get people following essentialist intuitions on board with value learning. I'll indirectly rely on some properties of morality that behave more like a pattern and less like an essence. I'd definitely be interested in feedback, and if you have moral essentialist intuitions, please forgive me for framing this post as talking about you, not to you.

1: Why bother?

AI is going to be the pivotal technology of the coming century. AI that does good things can help us manage existential risks like climate change, bio-engineered diseases, asteroids, or rogue AIs build by less careful people.

In a future where very clever AI does good things, everything is probably pretty great. In a future where very clever AI does things without any regard to their goodness, even if humanity doesn't get swept away to extinction, we're certainly not realizing the potential boon that AI represents.

Therefore, it would be really handy if you could program a computer to figure out what the right thing to do was, and then do it - or at least to make an honest try on both accounts. The status quo of humanity is not necessarily sustainable (see above about cli...
Blame games
by Benquo — 2019-05-06T02:38:12.868Z
Link: http://benjaminrosshoffman.com/blame-games/

In Excerpts from a larger discussion about simulacra, I worked through a well-known schema for distinguishing different relationships towards semantic reference, that are a natural result of interactions between shared-production games and expropriation games. Here, I analyze the coalition politics of such games.

The Survivor game

In zero-sum games, majoritarian decision rules (such as democracy) create an asymmetry - it's much easier to expropriate from a minority than from a majority - or, easier to transfer wealth to a majority than to a minority. Why would the majority vote for something they don't all benefit from?

A simple variant of this is the Survivor game, in which a single player is voted off the island at a time (see also the ancient Greek custom of ostracism). Since there's comparatively little advantage to being singled out for good, players will tend to want to avoid revealing information about themselves or their allies. Loudly voicing consensus opinion in ways that don't specify the implications for any person is fine because it's not informative. Anything that lets people distinguish you from the others is dangerous.

The idea of a Schelling point is that if players in a game need to converge on one location in a map, then in the absence of a strong incentive to favor one location, they will tend to converge on some obviously identifiable feature. For instance, in surveys, Thomas Schelling found that a surprisingly large number of people, if tasked with meeting someone on a specified day, in New York, with no further information, would converge on the information booth in Grand Central - and if no time was specified, they favored noon.

In a pure Survivor game, the first player to reveal their "location" loses. They become the feature everyone else converges on as an expropriation target. One natural side effect of this is coordination against any players who are narratively constrained by something other than the zero-sum game. For instance, if a widget-making group isn't under sharp performance pressure, anyone who's focused on actually making the widgets is going to have a hard time staying in lockstep with the group story, and is therefore the easiest target for expropriation and exclusion.

(Compare with Sarah Constantin's claim that group-coordination activities like dancing serve as a way to identify and exclude people who are out of sync wit...
Building trusted third parties
by anithite — 2019-05-05T23:18:27.786Z

How do two mutually distrusting entities(AGIs, space-faring societies, etc.) enact the results of a negotiated agreement when neither trusts the other? In cryptography there are Trusted third parties. This is a proposal for a protocol allowing them to be built.

This is done by having a replicator supplied by party A(Alice) self replicate. Party B(Bob) chooses one offspring from each generation to inspect destructively and another to continue the process. At each step Bob becomes twice as confident the remaining replicator is trustable. This is similar to a split and choose zero knowledge proof.

This protocol is carried out by smaller agents that Alice and Bob both trust. These entities must meet a few requirements.

They have to detect tampering

They have to hold secrets used for secure communication

They have to destroy the secrets if tampered with

How can these conditions fail?:

new physics is discovered allowing remote scanning or manipulation of matter

A very large nanotechnology gap between the two parties(IE:one is actually incompetent)

One knows femtotech the other doesn't (same as previous)

If these conditions cannot be met (the universe doesn't allow for them) then this isn't going to work and more generally it isn't possible to build trustable infrastructure. That's it. Secrets are used to secure communication with other agents that are part of or created by Alice or Bob. The physics we know about seems to allow this.

If one party has enough of a technological advantage they would just subsume the other rather than negotiate. Also, if precognition or time travel is possible the coin flip step of the protocol fails.

Implementation concerns

Alice would likely send more than just the replicator. Additional monitoring of the process might be necessary depending on the quality of tamper detection that is possible. Carrying out the process in space far from large masses that Bob could use to hide nefarious things seems like a sensible precaution.

The replicator might be designed to do mostly inspection and assembly rather than fabrication of components. This makes the replicator smaller and the replication process faster. A preferred embodiment is the use of thin sheet form components that can be fed into the replicator and inspected by nano-scale probing.

Bob can attack the process by tampering with the replicator or the materials he provides to it. For a

...
How much do major foundations grant per hour of staff time?
by Raemon — 2019-05-05T19:57:42.756Z

[Edit: original source has been found. The way I framed this question was off, since I'd mis-remembered it as "amount that a given grantmaker grants per year" rather than "grants per hour of staff time". Updated the title]

I recall reading an article once that claimed that, when examining many small and large foundations, it turned out that there was a maximum amount that a given grantmaker typically gave out. And as organizations scaled to give out more money, this amount stayed surprisingly fixed, with a higher overhead ratio than you might have expected.

(i.e. when an org gives out a million a year, it has N grantmakers, and when it gives out 100 million a year, it typically has 100N grantmakers).

I don't remember the number, or the methodology that determined it. Curious if anyone can remember the article. (It might have been from OpenPhil's blog, or it might have been some random news site).

I vaguely remember the number "3" being involved, possibly $300k, or $3 million.

The takeaway I remember was something like "you might naively think you can scale up an organization and then give away money more efficiently, but weird forces seem to limit that."

Does this sound familiar to anyone?
The why and how of daily updates
by VipulNaik — 2019-05-05T15:21:42.300Z

In August 2018, my work colleague Tomo Kazahaya and I came up with the
idea of using GitHub issues to keep track of daily work checklists. We
created a private GitHub repository called daily-updates. In
October, I copied over the idea to managing personal checklists; you
can see my first issue created October
20 or my issue
for yesterday.

Since then, I've been creating daily updates issues on my personal
repository on any days where I'm doing meaningful work on personal
projects or tasks beyond daily personal maintenance.

In this post, I'll talk briefly about the following:

The benefits of a broadly visible written document for a personal checklist

The benefits of daily granularity for a personal checklist

The benefits of using GitHub issues to record daily updates

Some protocol questions I don't have clear answers to

The benefits of a broadly visible written document for a personal checklist

I'm using the term "broadly visible" instead of public, because in
some cases (such as companies) daily updates of employees refer to a
lot of confidential company information, and therefore can only be
visible within the company. In this context, "broadly visible" would
mean visible within the company.

The emphasis on "broadly visible written document" is to contrast with
two other kinds of approaches:

Private personal checklists, that each person manages on their own,
but are not visible to others: The advantage of a broadly visible
document rather than a private personal checklist is that people can
check any time what the others are working on. This allows them to
offer thoughts and feedback, and also gives a better
organization-wide picture of what is going on.

Daily stand-ups, where everybody describes what they plan to do
during the day: Written documents are preferable to daily stand-ups
because they can be continuously updated as needed, can be
referenced any time, and work better for teams that don't all start
work at the same time. Also, my historical experience has been that
most people hate stand-ups; the Internet
seems
to
agree.

A dilemma that sometimes comes up is that some subset of one's daily
tasks needs to be kept private, and therefore cannot be explicitly
included in a broadly visible daily checklist. I've encountered this
sort of situation both in my work checklists and non-work checklists.

For these kinds of situations, I've generally found it

...
A reckless introduction to Hindley-Milner type inference
by philh — 2019-05-05T14:00:00.862Z

(I've been editing this post on and off for almost a year. I'm not really happy with it, but I suspect I never will be.)

Note: the math formatting got lost in crossposting to LW. With the help of the mods, I've mostly cleared it up, but you may wish to read on my blog instead.

Several months ago I gave a talk at work about Hindley-Milner type inference. When I agreed to give the talk I didn't know much about the subject, so I learned about it. And now I'm writing about it, based on the contents of my talk but more fleshed out and hopefully better explained.

I call this a reckless introduction, because my main source is wikipedia. A bunch of people on the internet have collectively attempted to synthesise a technical subject. I've read their synthesis, and now I'm trying to re-synthesise it, without particularly putting in the effort to check my own understanding. I'm not going to argue that this is a good idea. Let's just roll with it.

I'm also trying to tie in some quasi-philosophy that surely isn't original to me but I don't know if or where I've encountered it before.1

Background

When people write software, sometimes it doesn't do exactly what we want. One way to find out is to try running it and see, but that's not ideal because any complicated program will have way too many possible inputs to test. (Especially when you consider that inputs include things like "amount of free space on disk" and "time taken for a web server to respond to a request".) So it would be nice if we could mathematically prove whether our software does what we want, without actually running it. Can we do that?

That's not a very well-defined question, but we can ask more precise versions of it. Here's a well-known one: given some possible input to our software, we might want to prove that our software will eventually stop running. Can we prove that?

That question is known as the halting problem, and the simple answer is that we can't, not in general; the halting problem is undecideable. But the full answer is more complicated.

To solve the halting problem, we want a program that, when shown another program and some input to be fed to that program, satisfies three different conditions:

It will always return an answer.

The answer will always be either "yes, this always terminates" or "no, sometimes this doesn't terminate".

The answer is always correct.

And that's not possible. But

...
Gwern's "Why Tool AIs Want to Be Agent AIs: The Power of Agency" 
by habryka — 2019-05-05T05:11:45.805Z
Link: https://www.gwern.net/Tool-AI

I somehow hadn't read this post until now, so I am posting this here in case I am not the only one (and I wasn't able to find a previous linkpost for it). Relevant to relatively recent discussion on AI-as-a-service, but also just good as a broad reference.
Is value drift net-positive, net-negative, or neither?
by MarisaJurczyk — 2019-05-05T02:37:40.880Z

Crossposted from the EA forum

I've been asked variations of this question a few times recently, as I'm studying value drift for my undergraduate thesis, so I thought I would seek out others' thoughts on this.

I suppose part of this depends on how we define value drift. I've seen value drift defined as broadly as changes in values (from the Global Optimum podcast) and as narrowly as becoming less motivated to do altruistic things over time (from Joey Savoie's forum post). While the latter seems almost certainly net-negative, how the former plays out is a little less clear to me.

This leads me to wonder if there might be different kinds of value drift that may be varying degrees of good or bad.

Thoughts?
[AN #55] Regulatory markets and international standards as a means of ensuring beneficial AI
by Rohin Shah — 2019-05-05T02:20:01.030Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter.

The improvements to the newsletter continue! Rob Miles has generously volunteered to make the

Alignment Newsletter Podcast

. Chances are that the podcast will trail a week behind the emails, unless I manage to get my act together and give Rob a preview of the newsletter in advance.

Highlights

Standards for AI Governance: International Standards to Enable Global Coordination in AI Research & Development

(Peter Cihon): This technical report argues that we can have an outsized impact on the future of AI by influencing standards on AI so that they help ensure that AI systems are safe and beneficial, in addition to making the deployment of AI more efficient. A standard here could be a product like Tensorflow or Gym, or a process like

this list

. It's particularly useful to focus on international standards: since corporations can simply leave the country to escape national regulations, there is a race to the bottom on the stringency of national standards, and so they can't effect as much change.

It may be particularly valuable to influence existing organizations that set standards because they are very responsive to expert opinion. It is also possible to develop a standard privately, and then "convert" it into an international standard. (This happened with the C programming language and the PDF file format.) Such influence can be used to change the culture around AI development, e.g. to put safety more at the forefront.

Rohin's opinion: I would guess that the most influential standards are "network standards" like Tensorflow: they make it easier for everyone to develop AI systems. However, the benefit here is in having any standard at all, and so it seems unlikely that such standards could also effect a change in culture that's unrelated to the efficiency aspect of the standard. That said, the report convinced me that "enforced standards" are also impactful: even if the standard requires active enforcement to prevent organizations from ignoring it, organizations will often choose to comply with the standard in order to get a certification that builds consumer trust in them.

Regulatory Markets for AI Safety

(Jack Clark et al): This paper presents an idea o...
Should Effective Altruism be at war with North Korea?
by Benquo — 2019-05-05T01:50:15.218Z
Link: http://benjaminrosshoffman.com/should-effective-altruism-be-at-war-with-north-korea/

Summary: Political constraints cause supposedly objective technocratic deliberations to adopt frames that any reasonable third party would interpret as picking a side. I explore the case of North Korea in the context of nuclear disarmament rhetoric as an illustrative example of the general trend, and claim that people and institutions can make better choices and generate better options by modeling this dynamic explicitly. In particular, Effective Altruism and academic Utilitarianism can plausibly claim to be the British Empire's central decisionmaking mechanism, and as such, has more options than its current story can consider.

Context

I wrote to my friend Georgia in response to this Tumblr post.

Asymmetric disarmament rhetoric

Ben: It feels increasingly sketchy to me to call tiny countries surrounded by hostile regimes "threatening" for developing nuclear capacity, when US official policy for decades has been to threaten the world with nuclear genocide.

Strong recommendation to read Daniel Ellsberg's The Doomsday Machine.

Georgia: Book review: The Doomsday Machine

So I get that the US' nuclear policy was and probably is a nightmare that's repeatedly skirted apocalypse. That doesn't make North Korea's program better.

Ben [feeling pretty sheepish, having just strongly recommended a book my friend just reviewed on her blog]: "Threatening" just seems like a really weird word for it. This isn't about whether things cause local harm in expectation - it's about the frame in which agents trying to organize to defend themselves are the aggressors, rather than the agent insisting on global domination.

Georgia: I agree that it's not the best word to describe it. I do mean "threatening the global peace" or something rather than "threatening to the US as an entity." But, I do in fact think that North Korea building nukes is pretty aggressive. (The US is too, for sure!)

Maybe North Korea would feel less need to defend itself from other large countries if it weren't a literal dictatorship - being an oppressive dictatorship with nukes is strictly worse.

Ben: What's the underlying thing you're modeling, such that you need a term like "aggression" or "threatening," and what role does it play in that model?

Georgia: Something like destabilizing to the global order and ...
Dishonest Update Reporting
by Zvi — 2019-05-04T14:10:00.742Z

Related to: Asymmetric Justice, Privacy, Blackmail

Previously (Paul Christiano): Epistemic Incentives and Sluggish Updating

The starting context here is the problem of what Paul calls sluggish updating. Bob is asked to predict the probability of a recession this summer. He said 75% in January, and how believes 50% in February. What to do? Paul sees Bob as thinking roughly this:

If I stick to my guns with 75%, then I still have a 50-50 chance of looking smarter than Alice when a recession occurs. If I waffle and say 50%, then I won’t get any credit even if my initial prediction was good. Of course if I stick with 75% now and only go down to 50% later then I’ll get dinged for making a bad prediction right now—but that’s little worse than what people will think of me immediately if I waffle.

Paul concludes that this is likely:

Bob’s optimal strategy depends on exactly how people are evaluating him. If they care exclusively about evaluating his performance in January then he should always stick with his original guess of 75%. If they care exclusively about evaluating his performance in February then he should go straight to 50%. In the more realistic case where they care about both, his optimal strategy is somewhere in between. He might update to 70% this week.

This results in a pattern of “sluggish” updating in a predictable direction: once I see Bob adjust his probability from 75% down to 70%, I expect that his “real” estimate is lower still. In expectation, his probability is going to keep going down in subsequent months. (Though it’s not a sure thing—the whole point of Bob’s behavior is to hold out hope that his original estimate will turn out to be reasonable and he can save face.)

This isn’t ‘sluggish’ updating, of the type we talk about when we discuss the Aumann Agreement Theorem and its claim that rational parties can’t agree to disagree. It’s dishonest update reporting. As Paul says, explicitly.

I think this kind of sluggish updating is quite common—if I see Bob assign 70% probability to something and Alice assign 50% probability, I expect their probabilities to gradually inch towards one another rather than making a big jump. (If Alice and Bob were epistemically rational and honest, their probabilities would immediately take big enough jumps that we wouldn’t be able to predict in advance who will end up with the higher number. Needless to say, this is no

...