trentmkelly/LessWrong-43k huggingface dataset:
Alignment Newsletter #36
by Rohin Shah — 2018-12-12T01:10:01.398Z

Developing a theory of values to solve extrapolation issues, and an approach to train AI systems to reason well

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

Highlights

Why we need a theory of human values (Stuart Armstrong): There are many different sources of information for human values, such as behavior, speech, facial expressions/emotions, and extrapolations of what a human would do. These have all been honed to produce similar preferences in our current environment. However, the environment will change a lot due to the AI's actions, inducing a distributional shift, after which we should no longer expect the values inferred from these different methods to agree with each other. In addition, we also have the problem that each method only applies in some circumstances -- for example, people are likely to misrepresent their values if asked in a courtroom. We could try to patch these problems by having a meta-method that chooses from the various methods of value learning, as well as predicting human judgments about when each method applies. However, then we'd likely have similar issues with the meta-method and predictions, which are also likely to be specific to the current environment. Instead, we should have a theory of human values, from which we can get principled approaches to resolve these problems.

Rohin's opinion: I strongly agree that due to the problems mentioned in this post, we shouldn't be trying to mix and match value learning methods to infer a static notion of human preferences that is then optimized over the long term. I don't personally work on building a theory of human values because it seems like I could apply the same critiques to the result: the theory is specialized to our current situation and won't capture changes in the future. But of course this is hard to predict without knowing the theory. My preferred solution is not to build a system that is optimizing a goal-directed utility function over the long term, and to find other ways of making an AI system that are still just as useful.

Technical AI alignment

Iterated amplification sequence

Factored Cognition (Andreas Stuhlmüller): This was previously summarized in AN #12. While this post describes a project from Ought, it explains many of t...
A hundred Shakespeares
by Stuart_Armstrong — 2018-12-11T23:11:48.668Z

In his post on science slowing down, Scott said:

"Are there a hundred Shakespeare-equivalents around today? This is a harder problem than it seems – Shakespeare has become so venerable with historical hindsight that maybe nobody would acknowledge a Shakespeare-level master today even if they existed – but still, a hundred Shakespeares?"

I'd argue that there are way more than a hundred Shakespeares around today, and there were several in Shakespeare's time. By Shakespeares, I mean authors who could have produced works of comparable quality to Shakespeare, by some reasonable measure of quality.

This seems surprising; there do not seem to be hundred living authors that are almost universally agreed to be must-reads in the same way that Shakespeare was.

But this lack hints at a resolution of the paradox: we just don't have space for a hundred authors with the same fervour as we make space for Shakespeare. Neither as individuals nor as cultures can we fit these in. Shakespeare was a literary superstar. And superstars are rare, due to network effects and the power law of fame.

So my thesis would be that:

There are many non-superstars who could plausibly have become superstars, and if they had done, they would produce works of comparable quality to the superstars.

Part of this is the halo effect: superstars just get judged as better than anyone else.

Also, just by being famous, the interpretation of their work is altered. Bits of Shakespeare have permeated popular culture, and many articles and theories have been created about him. When we watch a Shakespeare play, we don't just see the words; we see the layers of cultural meaning and interpretation that have accumulated on it.

I'd argue that, just by knowing that a play is by Shakespeare, we assume that it's deep and meaningful, and read in deeper interpretations and symbolism than we would otherwise. If we rediscovered two old plays, and they were word for word identical, but one was believed to be by Shakespeare and the other by some forgotten minor playwright, I'd expect that the first one would be a better play, just by what the audience would bring to it.

Apart from those effects, superstars have the unique ability to focus more on their own vision. They have great self-confidence, and they can afford to trust that their audiences will have the patience to follow them where they want to go - rather than expecti

...
Norms of Membership for Voluntary Groups
by sarahconstantin — 2018-12-11T22:10:00.975Z

Epistemic Status: Idea Generation

One feature of the internet that we haven’t fully adapted to yet is that it’s trivial to create voluntary groups for discussion. It’s as easy as making a mailing list, group chat, Facebook group, Discord server, Slack channel, etc.

What we don’t seem to have is a good practical language for talking about norms on these mini-groups — what kind of moderation do we use, how do we admit and expel members, what kinds of governance structures do we create.

Maybe this is a minor thing to talk about, but I suspect it has broader impact. In past decades voluntary membership in organizations has declined in the US — we’re less likely to be members of the Elks or of churches or bowling leagues — so lots of people who don’t have any experience in founding or participating in traditional types of voluntary organizations are now finding themselves engaged in governance without even knowing that’s what they’re doing.

When we do this badly, we get “internet drama.” When we do it really badly, we get harassment campaigns and calls for regulation/moderation at the corporate or even governmental level. And that makes the news. It’s not inconceivable that Twitter moderation norms affect international relations, for instance.

It’s a traditional observation about 19th century America that Americans were eager joiners of voluntary groups, and that these groups were practice for democratic participation. Political wonks today lament the lack of civic participation and loss of trust in our national and democratic institutions. Now, maybe you’ve moved on; maybe you’re a creature of the 21st century and you’re not hoping to restore trust in the institutions of the 20th. But what will be the institutions of the future? That may well be affected by what formats and frames for group membership people are used to at the small scale.

It’s also relevant for the future of freedom. It’s starting to be a common claim that “give people absolute ‘free speech’ and the results are awful; therefore we need regulation/governance at the corporate or national level.” If you’re not satisfied with that solution (as I’m not), you have work to do — there are a lot of questions to unpack like “what kind of ‘freedom’, with what implementational details, is the valuable kind?”, “if small-scale voluntary organizations can handle some of the functions of the state, how exact...
Quantum immortality: Is decline of measure compensated by merging timelines?
by avturchin — 2018-12-11T19:39:28.534Z

I wrote an article about the quantum immortality which, I know, is a controversial topic, and I would like to get comments on it. The interesting twist, suggested in the article, is the idea of measure increase which could compensate declining measure in quantum immortality. (There are other topics in the article, like the history of QM, its relation to the multiverse immortality, the utility of cryonics, impossibility of euthanasia and the relation of QI to different decision theories.)

The standard argument against quantum immortality in MWI runs as following. One should calculate the expected utility by multiplying the expected gain on the measure of existence (roughly equal to the one's share of the world’s timelines). In that case, if someone expects to win 10.000 USD in the Quantum suicide lottery with 0.01 chance of survival, her actual expected utility is 100 USD (ignoring negutility of death). So, the rule of thumb is that the measure declines very quickly after series of quantum suicide experiments, and thus this improbable timeline should be ignored. The following equation could be used for U(total) = mU, where m is measure and U is expected win in the lottery.

However, if everything possible exists in the multiverse, there are many my pseudo-copies, which differ from me in a few bits, for example, they have a different phone number or different random child memory. The difference is small but just enough for not regard them as my copies.

Imagine that this different child memory is 1kb (if compressed) size. Now, one morning both me and all my pseudo-copies forget this memory, and all we become exactly the same copies. In some sense, our timelines merged. This could be interpreted as a jump in my measure, which will as high as 2power1024 = (roughly) 10E300. If I use the equation U(total) = mU I can get an extreme jump of my utility. For example, I have 100 USD and now my measure increased trillion of trillion of times, I supposedly get the same utility as if I become mega-multi-trillioner.

As a result of this absurd conclusion, I can spend the evening hitting my head with a stone and thus losing more and more memories, and getting higher and higher measure, which is obviously absurd behaviour for a human being - but could be a failure mode for an AI, which uses the equation to calculate the expected utility.

In case of the Quantum suici...
Bounded rationality abounds in models, not explicitly defined
by Stuart_Armstrong — 2018-12-11T19:34:17.476Z

Last night, I did not register a patent to cure all forms of cancer. Even though it’s probably possible to figure such a cure out, from basic physics and maybe a download of easily available biology research papers.

Can we then conclude that I don’t want cancer to be cured – or, alternatively, that I am pathologically modest and shy, and thus don’t want the money and fame that would accrue?

No. The correct and obvious answer is that I am boundedly rational. And though an unboundedly rational agent – and maybe a superintelligence – could figure out a cure for cancer from first principles, poor limited me certainly can’t.

Modelling bounded rationality is tricky, and it is often accomplished by artificially limiting the action set/action space. Many economic models use revealed preferences, and feature agents that are assumed to be fully rational, but who are restricted to choosing between a tiny set of possible goods or lotteries. They don’t have the options of developing new technologies, rousing the population to rebellion, going online and fishing around for functional substitutes, founding new political movements, begging, befriending people who already have the desired goods, setting up GoFundMe pages, and so on.

There’s nothing wrong with modelling bounded rationality via action set restriction, as long as we’re aware of what we’re doing. In particular, we can’t naively conclude that because a such a model fits with observation, that therefore humans actually are fully rational agents. In particular, though economists are right that humans are more rational than we might naively suppose, thinking of us as rational, or “mostly rational”, is a colossally erroneous way of thinking. In terms of achieving our goals, as compared with a rational agent, we are barely above agents acting randomly.

Another problem with using small action sets, is that it may lead us to think that an AI might be similarly restricted. That is unlikely to be the case; an intelligent robot walking around would certainly have access to actions that no human would, and possibly ones we couldn’t easily imagine.

Finally, though action set reduction can work well in toy models, it is wrong about the world and about humans. So as we make more and more sophisticated models, there will come a time when we have to discard it, and tackle head-on the difficult issue of defining bounded rationality pro...
Figuring out what Alice wants: non-human Alice
by Stuart_Armstrong — 2018-12-11T19:31:13.830Z

I’ve shown that we cannot deduce the preferences of a potentially irrational agent. Even simplicity priors don’t help. We need to make extra ‘normative’ assumptions in order to be able to say anything about these preferences.

I then presented a more intuitive example, in which Alice was playing poker, and had two possible beliefs about Bob’s hand, and two possible preferences: wanting money, or wanting Bob (which, in that situations, translated into wanting to lose to Bob).

That example illustrated the impossibility result, within the narrow confines of that situation – if Alice calls, she could be a money-maximiser expecting to win, or a love-maximiser expecting to lose.

As has been pointed out, this uncertainty doesn’t really persist if we move beyond the initial situation. If Alice was motivated by love or money, we would expect to be able to tell which one, by seeing what she does in other situations – how does she respond to Bob’s flirtations, what does she confess to her closest friends, how does she act if she catches a peek of Bob’s cards, etc…

So if we look at her more general behaviour, it seems that we have two possible versions of Alice. First, Am, who clearly wants money, and A♡, who clearly wants Bob. The actions of these two agents match up in the specific case I described, but not in general. Doesn’t this undermine my claim that we can’t tell the preferences of an agent from their actions?

What’s actually happening here is that we’re already making a lot of extra assumptions when we’re interpreting Am or A♡’s actions. We model other humans in very specific and narrow ways, and other humans do the same – and their models are very similar to ours (consider how often humans agree that another human is angry, or that being drunk impairs rationality). The agreement isn’t perfect, but is much better than random.

If we set those assumptions aside, then we can see what the theorem implies. There is a possible agent A′m, whose preference is for love, but that nevertheless acts identically to Am (and the reverse for money-loving A′♡ versus A♡). A′m and A′♡ are perfectly plausible agents – they just aren’t ‘human’ according to our models of what being human means.

It’s because of this that I’m somewhat optimistic we can solve the value learning problem, and why I often say the problem is “impossible in theory, but doable in practice”. Humans make a whole host

...
Assuming we've solved X, could we do Y...
by Stuart_Armstrong — 2018-12-11T18:13:56.021Z

The year is 1933. Leó Szilárd has just hypothised the nuclear chain reaction. Worried researchers from proto-MIRI or proto-FHI ask themselves "assuming we've solved the issue of nuclear chain reactions in practice, could we build a nuclear bomb out of it"?

Well, what do we mean by "assuming we've solved the issue of nuclear chain reactions"? Does it mean that "we have some detailed plans for viable nuclear bombs, including all the calculations needed to make them work, and everything in the plans is doable by a rich industrial state"? In that case, the answer to "could we build a nuclear bomb out of it?" is a simple and trivial yes.

Alternatively, are we simply assuming "there exists a collection of matter that supports a chain reaction"? In which case, note that the assumption is (almost) completely useless. In order to figure out whether a nuclear bomb is buildable, we still need to figure out all the details of chain reactions - that assumption has bought us nothing.

Assuming human values...

At the recent AI safety unconference, David Krueger wanted to test, empirically, whether debate methods could be used for creating aligned AIs. At some point in the discussion, he said "let's assume the question of defining human values is solved", wanting to move on to whether a debate-based AI could then safely implement it.

But as above, when we assume that an underdefined definition problem (human values) is solved, we have to be very careful what we mean - the assumption might be useless, or might be too strong, and end up solving the implementation problem entirely.

In the conversation with David, we were imagining a definition of human values related to what humans would answer if we could reflexively ponder specific questions for thousands of years. One could object to that definition on the grounds that people can be coerced or tricked into giving the answers that the AI might want - hence the circumstances of that pondering is critical.

If we assume X="human values are defined in this way", could an AI safely implement X via debate methods? Well, what about coercion and trickery by the AI during the debate process? It could be that X doesn't help at all, because we still have to resolve all of the same issues.

Or, conversely, X might be too strong - it might define what trickery is, which solves a lot of the implementation problem for free. Or, in the extreme case

...
Who's welcome to our LessWrong meetups?
by ChristianKl — 2018-12-10T13:31:17.357Z

As part of announcing meetups publically, it's good to write in the meetup description about what kind of people would likely be a good match for the meetup. I still haven't gotten a good description myself.

How would you describe the kind of people we are in words that are clear to outsiders?
How Old is Smallpox?
by Raemon — 2018-12-10T10:50:33.960Z

The conventional view is that smallpox has been around since antiquity, but more recent evidence has suggested it's actually only around 500 years old.

So I have a research/rationality question: how conclusive is the "500 years old hypothesis"? I don't really have the expertise to evaluate it.

The wikipedia entry briefly notes the new findings, but doesn't seem to have rewritten the overall history section:

The earliest credible clinical evidence of smallpox is found in the smallpox-like disease in medical writings from ancient India (as early as 1500 BC),[54][55]Egyptian mummy of Ramses V who died more than 3000 years ago (1145 BC)[56] and China (1122 BC).[57] It has been speculated that Egyptian traders brought smallpox to India during the 1st millennium BC, where it remained as an endemic human disease for at least 2000 years. Smallpox was probably introduced into China during the 1st century AD from the southwest, and in the 6th century was carried from China to Japan.[26] In Japan, the epidemic of 735–737 is believed to have killed as much as one-third of the population.[14][58] At least seven religious deities have been specifically dedicated to smallpox, such as the god Sopona in the Yoruba religion. In India, the Hindu goddess of smallpox, Sitala Mata, was worshiped in temples throughout the country.[59]

A different viewpoint is that smallpox emerged 1588 AD and the earlier reported cases were incorrectly identified as smallpox.[60][61]

Paper: 17th Century Variola Virus Reveals the Recent History of Smallpox

The paper arguing the 500 years hypothesis is here.

Highlights:

• Variola virus genome was reconstructed from a 17th century mummified child

• The archival strain is basal to all 20th century strains, with same gene degradation

• Molecular-clock analyses show that much of variola virus evolution occurred recently

Abstract

Smallpox holds a unique position in the history of medicine. It was the first disease for which a vaccine was developed and remains the only human disease eradicated by vaccination. Although there have been claims of smallpox in Egypt, India, and China dating back millennia [1, 2, 3, 4], the timescale of emergence of the causative agent, variola virus (VARV), and how it evolved in the context of increasingly widespread immunization, have proven controversial [4, 5, 6, 7, 8, 9].

In particular, some molecular-clock-

...
Why should EA care about rationality (and vice-versa)?
by Gordon Seidoh Worley — 2018-12-09T22:03:58.158Z

There's a lot of overlap between the effective altruism movement and the LessWrong rationality movement in terms of their membership, but each also has many people who are part of one group and not the other. For those in the overlap, why should EA care about rationality and rationality care about EA?
Measly Meditation Measurements
by justinpombrio — 2018-12-09T20:54:46.781Z

A few months ago, I decided to start meditating regularly, around an hour a day. It seemed like a good opportunity to measure possible effects, so I asked for advice on what to measure. This post summarizes the results. In short, while the subjective effects of meditation were strong, the measurements didn't show anything. This is a fine place to stop reading; I'm mostly posting this because I promised to.

I did mindfulness meditation, as guided by The Mind Illuminated. My object of focus was typically my breath (while sitting), or my steps (as hiking).

What I Measured

About once a week, I did some online tasks either before or after meditating. These were the Go/NoGo and CuedAttention tasks on quantified-mind.com, and this psychomotor vigilance task.

I set up roughly once or twice daily pings from TagTime for experience sampling.

What I Measured

My performance on the tasks looked entirely random. It wasn't better or worse after meditating, and it didn't get better or worse over time.

I have no idea how to do experience sampling. I understand that some people have moods. I'm almost always in a neutral mood, and so wasn't sure what to put most of the time. Also, I'm apparently often away from my phone, and missed many (most?) pings.

What I Learned

The Mind Illuminated is as good of a guide as I hoped it would be.

A few measly months of meditation isn't going to change anything like your performance on reaction-time-like tasks.

A few measly months of meditation will give you a fascinating look into your own mind. It's not what you think. I'd say more, but I'm deeply confused and don't have a good model.

Meditation retreats are great. I went on a two-day one, whose format wasn't particularly well-suited for me, and even this had a large effect on my practice.