trentmkelly/LessWrong-43k huggingface dataset:
Towards formalizing universality
by paulfchristiano — 2019-01-13T20:39:21.726Z

(Cross-posted at ai-alignment.com)

The scalability of iterated amplification or debate seems to depend on whether large enough teams of humans can carry out arbitrarily complicated reasoning. Are these schemes “universal,” or are there kinds of reasoning that work but which humans fundamentally can’t understand?

This post defines the concept of “ascription universality,” which tries to capture the property that a question-answering system A is better-informed than any particular simpler computation C.

These parallel posts explain why I believe that the alignment of iterated amplification largely depends on whether HCH is ascription universal. Ultimately I think that the “right” definition will be closely tied to the use we want to make of it, and so we should be refining this definition in parallel with exploring its applications.

I’m using the awkward term “ascription universality” partly to explicitly flag that this is a preliminary definition, and partly to reserve linguistic space for the better definitions that I’m optimistic will follow.

(Thanks to Geoffrey Irving for discussions about many of the ideas in this post.)

I. Definition

We will try to define what it means for a question-answering system A to be “ascription universal.”

1. Ascribing beliefs to A

Fix a language (e.g. English with arbitrarily big compound terms) in which we can represent questions and answers.

To ascribe beliefs to A, we ask it. If A(“are there infinitely many twin primes?”) = “probably, though it’s hard to be sure” then we ascribe that belief about twin primes to A.

This is not a general way of ascribing “belief.” This procedure wouldn’t capture the beliefs of a native Spanish speaker, or for someone who wasn’t answering questions honestly. But it can give us a sufficient condition, and is particularly useful for someone who wants to use A as part of an alignment scheme.

Even in this “straightforward” procedure there is a lot of subtlety. In some cases there are questions that we can’t articulate in our language, but which (when combined with A’s other beliefs) have consequences that we can articulate. In this case, we can infer something about A’s beliefs from its answers to the questions that we can articulate.

2. Ascribing beliefs to arbitrary computations

We are interested in whether A “can understand everything that could be understood by someone.” To clarify this, we need to be more p...
When is CDT Dutch-Bookable?
by abramdemski — 2019-01-13T18:54:12.070Z

I gave a rough sketch of a Dutch Book against CDT, with an example. I expect it can be turned into a fairly general theorem. What setting should we use to formalize this? What are the necessary assumptions? What's the general formulation of the Dutch Book? Are there interesting cases where this fails? How does it relate to anthropic decision problems, and anthropic Dutch-books?
Directions and desiderata for AI alignment
by paulfchristiano — 2019-01-13T07:47:13.581Z

Note: This is the first post from part four: what needs doing of the sequence on iterated amplification. The fourth part of the sequence describes some of the black boxes in iterated amplification and discusses what we would need to do to fill in those boxes. I think these are some of the most important open questions in AI alignment.

In the first half of this post, I’ll discuss three research directions that I think are especially promising and relevant to AI alignment:

Reliability and robustness. Building ML systems which behave acceptably in the worst case rather than only on the training distribution.
Oversight / reward learning. Constructing objectives and training strategies which lead our policies to do what we intend.
Deliberation and amplification. Surpassing human performance without simultaneously abandoning human preferences.

I think that we have several angles of attack on each of these problems, and that solutions would significantly improve our ability to align AI. My current feeling is that these areas cover much of the key work that needs to be done.

In the second half of the post, I’ll discuss three desiderata that I think should guide research on alignment:

Secure. Our solutions should work acceptably even when the environment itself is under the influence of an adversary.
Competitive. Our solutions should impose minimal overhead, performance penalties, or restrictions compared to malign AI.
Scalable. Our solutions should continue to work well even when the underlying learning systems improve significantly.

I think that taking these requirements seriously leads us to substantially narrow our focus.

It may turn out that these desiderata are impossible to meet, but if so I think that the first order of business should be understanding clearly why they are impossible. This would let us better target our work on alignment and better prepare for a future where we won’t have a completely satisfying solution to alignment.

(The ideas in this post are not novel. My claimed contribution is merely collecting these things together. I will link to my own writing on each topic in large part because that’s what I know.)

I. Research directions

1. Reliability and robustness

Traditional ML algorithms optimize a model or policy to perform well on the training distribution. These models can behave arbitrarily badly when we move away from the training distribution. Similarly...
What are the open problems in Human Rationality?
by Raemon — 2019-01-13T04:46:38.581Z

LessWrong has been around for 10+ years, CFAR's been at work for around 6, and I think there have been at least a few other groups or individuals working on what I think of as the "Human Rationality Project."

I'm interested, especially from people who have invested significant time in attempting to push the rationality project forward, what they consider the major open questions facing the field. (More details in this comment)

"What is the Rationality Project?"

I'd prefer to leave "Rationality Project" somewhat vague, but I'd roughly summarize it as "the study of how to have optimal beliefs and make optimal decisions while running on human wetware."

If you have your own sense of what this means or should mean, feel free to use that in your answer. But some bits of context for a few possible avenues you could interpret this through:

Early LessWrong focused a lot of cognitive biases and how to account for them, as well as Bayesian epistemology.

CFAR (to my knowledge, roughly) started from a similar vantage point and eventually started moving in the direction of "how to do you figure out what you actually want and bring yourself into 'internal alignment' when you want multiple things, and/or different parts of you want different things and are working at cross purposes. It also looked a lot into Double Crux, as a tool to help people disagree more productively.

CFAR and Leverage both ended up exploring introspection as a tool.

Forecasting as a field has matured a bit. We have the Good Judgment project.

Behavioral Economics has begun to develop as a field.

I recently read "How to Measure Anything", and was somewhat struck at how it tackled prediction, calibration and determining key uncertainties in a fairly rigorous, professionalized fashion. I could imagine an alternate history of LessWrong that had emphasized this more strongly.

With this vague constellation of organizations and research areas, gesturing at an overall field...

...what are the big open questions the field of Human Rationality needs to answer, in order to help people have more accurate beliefs and/or make better decisions?
Dutch-Booking CDT
by abramdemski — 2019-01-13T00:10:07.941Z

[This post is now superseded by a much better version of the argument.]

In a previous post, I speculated that you might be able to Dutch-Book CDT agents if their counterfactual expectations differed from the conditional expectations of EDT. The answer turns out to be yes.

I'm going to make this a short note rather than being very rigorous about the set of decision problems for which this works.

(This is an edited version of an email, and benefits from correspondence with Caspar Oesterheld, Gerard Roth, and Alex Appel. In particular, Caspar Oesterheld is working on similar ideas. My views on how to interpret the situation have changed since I originally wrote these words, but I'll save that for a future post.)

Suppose a CDT agent has causal expectations which differ from its evidential expectations, in a specific decision.

We can modify the decision by allowing an agent to bet on outcomes in the same act. Because the bet is made simultaneously with the decision, the CDT agent uses causal expected value, and will bet accordingly.

Then, immediately after (before any new observations come in), we offer a new bet about the outcome. The agent will now bet based on its evidential expectations, since the causal intervention has already been made.

For example, take a CDT agent in Death in Damascus. A CDT agent will take each action with 50% probability, and its causal expectations expect to escape death with 50% probability. We can expand the set of possible actions from (stay, run) to (stay, run, stay and make side bet, run and make side bet). The side bet could cost 1 util and pay out 3 utils if the agent doesn't die. Then, immediately after taking the action but before anything else happens, we offer another deal: the agent can get .5 util in exchange for -3 util conditional on not dying. We offer the new bet regardless of whether the agent agrees to the first bet.

The CDT agent will happily make the bet, since the expected utility is calculated along with the intervention. Then, it will happily sell the bet back, because after taking its action, it sees no chance of the 3 util payout.

The CDT agent makes the initial bet even though it knows it will later reverse the transaction at a cost to itself, because we offer the second transaction whether the agent agrees to the first or not. So, from the perspective of the initial decision, taking the bet is still +...
Time Biases
by PeterMcCluskey — 2019-01-12T21:35:54.276Z
Link: http://www.bayesianinvestor.com/blog/index.php/2019/01/10/time-biases/

Book review: Time Biases: A Theory of Rational Planning and Personal
Persistence, by Meghan Sullivan.

I was very unsure about whether this book would be worth reading, as it
could easily have been focused on complaints about behavior that experts
have long known are mistaken.

I was pleasantly surprised when it quickly got to some of the really
hard questions, and was thoughtful about what questions deserved
attention. I disagree with enough of Sullivan's premises that I have
significant disagreements with her conclusions. Yet her reasoning is
usually good enough that I'm unsure what to make of our disagreements -
they're typically due to differences of intuition that she admits are
controversial.

I had hoped for some discussion of ethics (e.g. what discount rate to
use in evaluating climate change), whereas the book focuses purely on
prudential rationality (i.e. what's rational for a self-interested
person). Still, the discussion of prudential rationality covers most of
the issues that make the ethical choices hard.

Personal identity

A key issue is the nature of personal identity - does one's identity
change over time?

Sullivan starts this discussion by comparing a single-agent model to a
model with multiple distinct stages.

I initially found that part of the book confusing, as neither model
seems close to mine, and she seemed to present them as the only possible
models. Then somewhat later on in the book, I figured out that both
those models assumed that identity needed to stay constant for some
time, and that in the case of the multiple stage model, the change in
identity was discontinuous.

Sullivan prefers that kind of single-agent model, and therefore
concludes that we should adopt a time-neutral stance (i.e. she rejects
any time-discounting, and expects us to care as much about our distant
future selves as we care about our present selves).

Sullivan eventually compares that single-agent model to some models in
which a current self partially identifies with a future self, which
would allow for time-discounting of how much we care about a self.

The most interesting of those models involves some notion of
connectedness between current and future selves.

This feels like the obviously right approach to me. I expect it to work
something like the "me" of a decade ago or a decade in the future being
something I consider to be maybe 75% "me". I.e. I practice
time-discoun

...
[Speech] Worlds That Never Were
by mingyuan — 2019-01-12T19:53:51.241Z

This is the speech I gave at the 2018 Bay Area Secular Solstice. It was written to fit the bill of 'concrete visions of the future.' There is one small change here from the version I gave : the Solstice organizers had me change the second-to-last sentence of the speech, because the event centered around the idea of disasters that fall short of extinction, and rebuilding from the ashes. I've changed it back to be more true to what I think Solstice is about - the complete and irrecoverable disaster looming on the horizon, and the reality of our desperate efforts to avoid it.

There are lots of people on the Internet and on TV every day telling you how the world is going to dissolve into chaos at any moment. When you think about all the people and animals needlessly suffering, it’s easy to think that the world is terrible. When you read the news, or look at the current state of the art in preparedness for global catastrophic risks, it’s easy to think that the world is doomed, and feel hopeless. But that’s why we’re here. We are here – in this community and in this planetarium – because we believe that there is a future worth fighting for.

What does that future look like? That’s hard to say. You know how sometimes, when you look directly at a star in the night sky, it fades so that you can barely see it, but when you look a little to the side it’s bright again? That’s sort of how I feel about imagining the future. It’s slippery, hard to get a grasp on, sort of just out of reach. But for now, let’s try to picture it.

Imagine the world a few hundred years from now. Humanity has completed a process of great deliberation. We’ve overcome global coordination problems, so there’s no more poverty and no more war, and existential risks have shrunk to a millionth of a millionth of their current size. We have overcome physics and engineering challenges and have ships on their way to the distant reaches of the galaxy. We have overcome our biological limitations and no longer die so easily of disease or accident or old age.

If we sit with that, it still feels abstract, and far away. So try to imagine yourself in that future, even if how we imagine it today will never be how it really is.

Maybe you’re with your family on a spaceship, off to colonize the edge of the galaxy. Your faces are pressed to the window as you watch the Earth grow small against the blackness. ...
Modernization and arms control don’t have to be enemies.
by Gentzel — 2019-01-12T18:50:01.148Z

On a recent panel on nuclear modernization and arms control at Brookings, there were a few points made about the potential synergies between arms control and modernization.

While modernization and arms control seem contradictory, it really depends on what you are trying to do with each. For example, if you want to reduce the number of nuclear weapons the U.S. has in its stockpile, it may actually make sense to improve the ability of the U.S. to produce new weapons. Why would that be you might ask? It’s because current stockpiles are a hedge, a hedge to replace warheads that are too old and to buy time for a future ability to produce warheads again faster rates. Toward the end of the Bush administration, maintenance and production were so bad that officials were worried that nuclear testing would become necessary again to verify that weapons worked.

Stranger yet about this dynamic is that due to U.S. difficulties in production, one of the panelists argued Russia currently has an advantage at warhead production that may incent them to race with the U.S. in the short-run. What are the potential results of this? It may partially explain Russia’s pull-out of the INF treaty: Russian leadership may not have perceived much of a risk to pulling out given lack of current U.S. ability to keep up at warhead production, and a lack of public support in the U.S. for new weapons generally. With the development of new intermediate-range missiles, the Russians will be able to free up some of their longer range nuclear missiles to aim at the U.S. instead of closer targets. If New START isn’t extended in 2021, it doesn’t seem unimaginable that Russia may seek to gain negotiation advantage unconstrained by numbers limits, until the U.S. ramps up its production ability again.

While I am not sure I agree fully with the panel, an implication to be drawn from their arguments is that from an equilibrium of treaty compliance, maintaining the ability to race can disincentivize the other side from treaty violation: it increases the cost to the other side of gaining advantage, and that can be especially decisive if your side has an economic advantage. This just generally seems to be an instance of the idea that it is important to maintain leverage in negotiations.

Speaking more broadly than just about nuclear weapons: a lack of technological development by a leading country is not suf...
Comments on CAIS
by Richard_Ngo — 2019-01-12T15:20:22.133Z

Over the last few months I’ve talked with Eric Drexler a number of times about his Comprehensive AI Services (CAIS) model of AI development, and read most of his technical report on the topic. I think these are important ideas which are well worth engaging with, despite personally being skeptical about many of the conclusions. Below I’ve summarised what I see as the core components of Eric’s view, followed by some of own arguments. Note that these are only my personal opinions. I did make some changes to the summary based on Eric’s comments on early drafts, to better reflect his position - however, there are likely still ways I’ve misrepresented him. Also note that this was written before reading Rohin’s summary of the same report, although I do broadly agree with most of Rohin’s points.

One useful piece of context for this model is Eric's background in nanotechnology, and his advocacy for the development of nanotech as "atomically precise manufacturing" rather than self-replicating nanomachines. The relationship between these two frameworks has clear parallels with the relationship between CAIS and a recursively self-improving superintelligence.

The CAIS model:

The standard arguments in AI safety are concerned with the development of a single AGI agent doing open-ended optimisation. Before we build such an entity (if we do so at all), we will build AI services which each perform a bounded task with bounded resources, and which can be combined to achieve superhuman performance on a wide range of tasks.
AI services may or may not be “agents”. However, under CAIS there will be no entity optimising extremely hard towards its goals in the way that most AI safety researchers have been worrying about, because:

Each service will be relatively specialised and myopic (focused on current episodic performance, not maximisation over the whole future). This is true of basically all current AI applications, e.g. image classifiers or Google Translate.
Although rational agents can be proved equivalent to utility-maximisers, the same is not necessarily true of systems of rational agents. Most such systems are fundamentally different in structure from rational agents - for example, individual agents within the system can compete with or criticise each other. And since AI services aren’t “rational agents” in the first place, a system composed of them is even less l

...
this Website uses Bayesian networks to evaluate claims - RootClaim
by Yoav Ravid — 2019-01-12T14:46:43.860Z

I found out this website, RootClaim, about a year and a half ago (a bit after it was founded), much before finding LW. that's how i found out about Bayesian probability, and i remember getting crazy at the potential of it :)

So i now thought "surly someone posted about it on LW" - i searched, and found nothing. I wonder if there's a reason or it's just that somehow no one here heard about it.

Anyway, it uses Bayesian trees (a special case of Bayesian networks), to evaluate different hypothesis.

anyone can sumbit a topic. and then anyone can submit hypothesis, sources, and so on. everything is crowdsourced.

as a tool, it seems to have a lot of potential, and i even saw that Nissim Taleb (Author of the black swan) twitted about it favorably.

main problem is, it seems to be quite inactive, both the community (which is super small), and the website itself. the last update was in December 2017...

I would love to hear what you guys think about it, or the potential of something like it :)

here's the link again: https://www.rootclaim.com
Ambitious vs. narrow value learning
by paulfchristiano — 2019-01-12T06:18:21.747Z

(Re)Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin's note: The definition of narrow value learning in the previous post focused on the fact that the resulting behavior is limited to some domain. The definition in this post focuses on learning instrumental goals and values. While the definitions are different, I have used the same term for both because I believe that they are both pointing at the same underlying concept. (I do not know if Paul agrees.) I'm including this post to give a different perspective on what I mean by narrow value learning, before delving into conceptual ideas within narrow value learning.

Suppose I’m trying to build an AI system that “learns what I want” and helps me get it. I think that people sometimes use different interpretations of this goal. At two extremes of a spectrum of possible interpretations:

The AI learns my preferences over (very) long-term outcomes. If I were to die tomorrow, it could continue pursuing my goals without me; if humanity were to disappear tomorrow, it could rebuild the kind of civilization we would want; etc. The AI might pursue radically different subgoals than I would on the scale of months and years, if it thinks that those subgoals better achieve what I really want.
The AI learns the narrower subgoals and instrumental values I am pursuing. It learns that I am trying to schedule an appointment for Tuesday and that I want to avoid inconveniencing anyone, or that I am trying to fix a particular bug without introducing new problems, etc. It does not make any effort to pursue wildly different short-term goals than I would in order to better realize my long-term values, though it may help me correct some errors that I would be able to recognize as such.

I think that many researchers interested in AI safety per se mostly think about the former. I think that researchers with a more practical orientation mostly think about the latter.

The ambitious approach

The maximally ambitious approach has a natural theoretical appeal, but it also seems quite hard. It requires understanding human preferences in domains where humans are typically very uncertain, and where our answers to simple questions are often inconsistent, like how we should balance our own welfare with the welfare of others, or what kinds of activities we really want to pursue vs. enjoying in the moment. (It seems unlikel...