trentmkelly/LessWrong-43k huggingface dataset:
Beliefs at different timescales
by Nisan — 2018-11-04T20:10:59.223Z

Why is a chess game the opposite of an ideal gas? On short timescales an ideal gas is described by elastic collisions. And a single move in chess can be modeled by a policy network.

The difference is in long timescales: If we simulated elastic collisions for a long time, we'd end up with a complicated distribution over the microstates of the gas. But we can't run simulations for a long time, so we have to make do with the Boltzmann distribution, which is a lot less accurate.

Similarly, if we rolled out our policy network to get a distribution over chess game outcomes (win/loss/draw), we'd get the distribution of outcomes under self-play. But if we're observing a game between two players who are better players than us, we have access to a more accurate model based on their Elo ratings.

Can we formalize this? Suppose we're observing a chess game. Our beliefs about the next move are conditional probabilities of the form P1(xk+1|x0⋯xk), and our beliefs about the next n moves are conditional probabilities of the form Pn(xk+1⋯xk+n|x0⋯xk). We can transform beliefs of one type into the other using the operators

(ΠnP1)(xk+1⋯xk+n|x0⋯xk):=n−1∏i=0P1(xk+i+1|x0⋯xk+i)

(ΣnPn)(xk+1|x0⋯xk):=∑xk+2⋯∑xk+nPn(xk+1⋯xk+n|x0⋯xk)

If we're logically omniscient, we'll have ΠnP1=Pn and ΣnPn=P1. But in general we will not. A chess game is short enough that Πn is easy to compute, but Σn is too hard because it has exponentially many terms. So we can have a long-term model Pn that is more accurate than the rollout ΠnP1, and a short-term model P1 that is less accurate than ΣnPn. This is a sign that we're dealing with an intelligence: We can predict outcomes better than actions.

If instead of a chess game we're predicting an ideal gas, the relevant timescales are so long that we can't compute Πn or Σn. Our long-term thermodynamic model Pn is less accurate than a simulation ΠnP1. This is often a feature of reductionism: Complicated things can be reduced to simple things that can be modeled more accurately, although more slowly.

In general, we can have several models at different timescales, and Π and Σ operators connecting all the levels. For example, we might have a short-term model describing the physics of fundmental particles; a medium-term model describing a person's motor actions; and a long-term model describing what that person accomplishes over the course of a year. The medium-term model will

...
No Really, Why Aren't Rationalists Winning?
by Sailor Vulcan — 2018-11-04T18:11:15.586Z

Reply to Extreme Rationality: It's Not That Great, Extreme Rationality: It could Be Great, the Craft and the Community and Why Don't Rationalists Win?

I’m going to say something which might be extremely obvious in hindsight:

If LessWrong had originally been targeted at and introduced to an audience of competent business people and self-improvement health buffs instead of an audience of STEM specialists and Harry Potter fans, things would have been drastically different. Rationalists would be winning.

Right now, rationalists aren’t winning. Rationality helps us choose which charities to donate to, and as Scott Alexander pointed out in 2009 it gives clarity of mind benefits. However, as he also pointed out in the same article, rationality doesn't seem to be helping us win in individual career or interpersonal/social areas of life.

It’s been nearly ten years since then, and I have yet to see any sign that this fact has changed. I considered the possibility that I just hadn’t heard about other rationalists’ practical success due to having not become a rationalist until around 2015, or simply because no one was talking about their success. Then I realized that was silly. If rationalists had started winning, at least one person would have posted about it here on lesswrong.com. I recently spoke to Scott Alexander, and he said he still agreed with everything he said in his article.

So rationalists aren’t winning. Why not? The Bayesian Conspiracy podcast (if I recall correctly), proposed the following explanation in one of their episodes: that rationality can only help us improve a limited amount relative to where we started out. They predicted that people who started out at a lower level of life success/cognitive functioning/talent cannot outperform non-rationalists who started out at a sufficiently high level.

This argument is fundamentally a cop-out. When others win in places where we fail, it makes sense to ask, “How? What knowledge, skills, qualities or experience do they have which we don't? And how might we obtain the same knowledge, skills, qualities or experience?” To say that others are simply more innately talented than we are, and leave it at that, doesn't explain the mechanism behind their hypothesized greater rate of improvement after learning rationality. It tells us why but not...
New GreaterWrong feature: image zoom + image slideshows
by Said Achmiz — 2018-11-04T07:34:44.907Z

GreaterWrong now has a new feature: image zoom / slideshows.

Whenever a post contains images (such as this recent post about embedded world-models), you can click on any image…

… to show an enlarged, “lightbox” style view of it:

You can then page through all the images in the post with the arrow keys, or by clicking the next/previous image buttons at the left/right edges of the screen:

Use the scroll wheel to zoom in or out:

When zoomed in, you can drag around the screen to pan (just like Google Maps, etc.).

Hit the space bar to reset size/position.

Hit Escape to close the slideshow/lightbox.

Edited to add: You can also hit accesskey-L to start or resume the slideshow. (On Mac Chrome, for example, that’s Control-Option-L.)
The easy goal inference problem is still hard
by paulfchristiano — 2018-11-03T14:41:55.464Z

Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin’s note: In this post (original here), Paul Christiano analyzes the ambitious value learning approach. He considers a more general view of ambitious value learning where you infer preferences more generally (i.e. not necessarily in the form of a utility function), and you can ask the user about their preferences, but it’s fine to imagine that you infer a utility function from data and then optimize it. The key takeaway is that in order to infer preferences that can lead to superhuman performance, it is necessary to understand how humans are biased, which seems very hard to do even with infinite data.

One approach to the AI control problem goes like this:

Observe what the user of the system says and does.
Infer the user’s preferences.
Try to make the world better according to the user’s preference, perhaps while working alongside the user and asking clarifying questions.

This approach has the major advantage that we can begin empirical work today — we can actually build systems which observe user behavior, try to figure out what the user wants, and then help with that. There are many applications that people care about already, and we can set to work on making rich toy models.

It seems great to develop these capabilities in parallel with other AI progress, and to address whatever difficulties actually arise, as they arise. That is, in each domain where AI can act effectively, we’d like to ensure that AI can also act effectively in the service of goals inferred from users (and that this inference is good enough to support foreseeable applications).

This approach gives us a nice, concrete model of each difficulty we are trying to address. It also provides a relatively clear indicator of whether our ability to control AI lags behind our ability to build it. And by being technically interesting and economically meaningful now, it can help actually integrate AI control with AI practice.

Overall I think that this is a particularly promising angle on the AI safety problem.

Modeling imperfection

That said, I think that this approach rests on an optimistic assumption: that it’s possible to model a human as an imperfect rational agent, and to extract the real values which the human is imperfectly optimizing. Without this assumption, it seems like some additional ideas are necessary.

To isolate this challenge...
My Intuitions on Counterlogical Reasoning
by DragonGod — 2018-11-02T21:36:15.494Z

Related to: Embedded Decisions.

Epistemic Status

This is the fruit of me thinking on counterlogical reasoning for several minutes. I expect there's nothing revolutionary here, and not much of import, and this post is designed more to help me specify how I think counterlogical reasoning is like intuitively, with the hope of reflecting on those intuitions and refining them as I advance in mathematical maturity∗.

There's also a part of me that was intrigued by the problem of formalising counterlogical reasoning being difficult when humans reason counterlogically all the time, which is what motivated me taking a shot in the dark.

Acknowledgement

I want to thank Diffractor and Intent for taking the time to help me refine my intuitions on counterlogical reasoning. This post would not have materialised where it not for their feedback.

Introduction

Consider a deterministic agent α which is faced with a decision problem in which it has two available actions ({a1,a2}). For the particular problem α is facing, the output of its decision algorithm is a mathematical fact (as the algorithm is deterministic). However in deciding which action to pick, α may engage in reasoning of the form "If I chose a1 what would the consequences be, and how would they compare to the consequences were I to choose a2?". Such reasoning is termed counterfactual as we consider scenarios that are contrary to what is actually factual (α will choose only one of ({a1,a2}) and would not choose the other option. Furthermore, as the fact of interest in this case is a logical fact (the output of a deterministic algorithm is a fixed logical fact), it's a special kind of counterfactual problem called a counterlogical problem (borrowing Garrabant and Demski's terminology).

Getting counterlogical reasoning right is very important as humans routinely engage in such reasoning (it seems to be an intuitively desirable way to select actions or policies), yet it seems that it is something that is incredibly difficult to formalise. That humans seem to do sensible counterlogical reasoning makes me believe that sensible counterlogical reasoning is possible, so I reflected on that using myself as the subject of inquiry. In the rest of the post I will describe my findings/conclusions.

Logical Implication is Not Counterlogical Implication

I suppose it would help to define counterfactual im... ,
Meta-execution
by paulfchristiano — 2018-11-01T22:18:10.656Z

This post describes meta-execution, my current proposal for capability amplification and security amplification.

(Meta-execution is annotated functional programming + strong HCH + a level of indirection. It is implemented in the amplify module of my ALBA repository.)

Goal

We are given an efficient agent A which competently pursues some values. We’d like to use a bunch of copies of A in order to implement a more powerful and robust agent Meta(A) with the same values.

Outline

Our basic plan is to build a machine out of copies of the agent; instead of asking the agent to make a decision directly, we ask it to implement the decision-making process by answering a sequence of questions of the form “what should happen next?”

The basic object in meta-execution is a message, which consists of text along with pointers to other messages or to agents.

Each box is a message. A is an agent who can respond to queries like “which of X and Y is larger?”

We can represent arbitrarily large objects as giant trees of messages and agents.

Meta-execution first forms a tree representing the question “what should be done?” It then asks the agent A to perform a sequence of operations on the tree that eventually lead to an answer. Then it executes that answer.

The initial tree might look something like this:

If you can answer this question, you can implement an agent.

At any given time, an agent who is operating on this tree can only “see” a few messages: it can read the text of those messages, and see pointers like [red] and [blue]. Initially the agent can see only the root of the tree.

If you are an agent tasked with processing a message, there are a few basic operations you can perform. You specify the “targets” of the action by specifying pointers you want to follow:

Look at another part of the tree which is not currently visible.
Spawn a new agent, and see a pointer to that agent.
Send a message to an agent, and see its reply. You can compose a message by writing it out with sub-messages in parentheses (); for example, “What is the smallest element in (the list with first element [green] and remaining elements [purple]) according to [blue]” would produce the message in the first image above, if [green], [purple], [blue] had appropriate values.
Terminate the current computation by composing a reply

...
Discussion on the machine learning approach to AI safety
by Vika — 2018-11-01T20:54:39.195Z

(Cross-posted from personal blog.)

At this year's EA Global London conference, Jan Leike and I ran a discussion session on the machine learning approach to AI safety. We explored some of the assumptions and considerations that come up as we reflect on different research agendas. Slides for the discussion can be found here.

The discussion focused on two topics. The first topic examined assumptions made by the ML safety approach as a whole, based on the blog post Conceptual issues in AI safety: the paradigmatic gap. The second topic zoomed into specification problems, which both of us work on, and compared our approaches to these problems.

Assumptions in ML safety

The ML safety approach focuses on safety problems that can be expected to arise for advanced AI and can be investigated in current AI systems. This is distinct from the foundations approach, which considers safety problems that can be expected to arise for superintelligent AI, and develops theoretical approaches to understanding and solving these problems from first principles.

While testing on current systems provides a useful empirical feedback loop, there is a concern that the resulting solutions might not be relevant for more advanced systems, which could potentially be very different from current ones. The Paradigmatic Gap post made an analogy between trying to solve safety problems with general AI using today's systems as a model, and trying to solve safety problems with cars in the horse and buggy era. The horse to car transition is an example of a paradigm shift that renders many current issues irrelevant (e.g. horse waste and carcasses in the streets) and introduces new ones (e.g. air pollution). A paradigm shift of that scale in AI would be deep learning or reinforcement learning becoming obsolete.

If we imagine living in the horse carriage era, could we usefully consider possible safety issues in future transportation? We could invent brakes and stop lights to prevent collisions between horse carriages, or seat belts to protect people in those collisions, which would be relevant for cars too. Jan pointed out that we could consider a hypothetical scenario with super-fast horses and come up with corresponding safety measures (e.g. pedestrian-free highways). Of course, we might also consider possible negative effects on the human body from moving at high speeds, which turned out not to ...
What is ambitious value learning?
by Rohin Shah — 2018-11-01T16:20:27.865Z

I think of ambitious value learning as a proposed solution to the specification problem, which I define as the problem of defining the behavior that we would want to see from our AI system. I italicize “defining” to emphasize that this is not the problem of actually computing behavior that we want to see -- that’s the full AI safety problem. Here we are allowed to use hopelessly impractical schemes, as long as the resulting definition would allow us to in theory compute the behavior that an AI system would take, perhaps with assumptions like infinite computing power or arbitrarily many queries to a human. (Although we do prefer specifications that seem like they could admit an efficient implementation.) In terms of DeepMind’s classification, we are looking for a design specification that exactly matches the ideal specification. HCH and indirect normativity are examples of attempts at such specifications.

We will consider a model in which our AI system is maximizing the expected utility of some explicitly represented utility function that can depend on history. (It does not matter materially whether we consider utility functions or reward functions, as long as they can depend on history.) The utility function may be learned from data, or designed by hand, but it must be an explicit part of the AI that is then maximized.

I will not justify this model for now, but simply assume it by fiat and see where it takes us. I’ll note briefly that this model is often justified by the VNM utility theorem and AIXI, and as the natural idealization of reinforcement learning, which aims to maximize the expected sum of rewards, although typically rewards in RL depend only on states.

A lot of conceptual arguments, as well as experiences with specification gaming, suggest that we are unlikely to be able to simply think hard and write down a good specification, since even small errors in specifications can lead to bad results. However, machine learning is particularly good at narrowing down on the correct hypothesis among a vast space of possibilities using data, so perhaps we could determine a good specification from some suitably chosen source of data? This leads to the idea of ambitious value learning, where we learn an explicit utility function from human behavior for the AI to maximize.

This is very related to inverse reinforcement learning (IRL) in the machine learning literatu...
Implementations of immortality
by Richard_Ngo — 2018-11-01T14:20:01.494Z
Link: http://thinkingcomplete.blogspot.com/2018/04/implementations-of-immortality.html

(Inspired by Eliezer's essays on designing utopias.)

I was recently talking to a friend about what key features society would need for people to be happy throughout arbitrarily-long lives - in other words, what would a utopia for immortals look like? He said that it would require continual novelty. But I think that by itself, novelty is under-specified. Almost every game of Go ever played is novel and unique, but eventually playing Go would get boring. Then you could try chess, I suppose - but at a certain point the whole concept of board games would become tiresome. I don't think bouncing around between many types of different activities would be much better, in the long run. Rather, the sort of novelty that's most desirable is a change of perspective, such that you find meaning in things you didn't appreciate before. That interpretation of novelty is actually fairly similar to my answer; I said that the most important requirement is a feeling of progress. By this I mean:

Your past isn't being lost as it recedes from you.
Your future will be better than your past - in qualitative ways as well as quantitative.
You receive increasing social recognition for your achievements.
You feel like you are continually growing as a person.

Some of these criteria are amenable to technical solutions - for example, memory enhancements would be very helpful for the first. But simply abiding by the basic principles of liberalism makes others very difficult to universalise. As long as we allow people to make their own choices, there will be some people who end up falling into addiction, or lethargy, or self-destructive spirals. We could theoretically make this rarer by having stronger social or legal norms, so that people still have freedom, but not total freedom. We could also make the consequences of failure less unpleasant (e.g. with welfare systems) and less permanent (e.g. by eradicating the most addictive drugs). Yet even then, the social repercussions of being unsuccessful would still weigh on people heavily - man does not live by bread alone, but by the status judgements of his peers.

Fortunately, perceived social status is not zero-sum, because different subcultures value different things. That helps many more people receive social recognition for their achievements; ideally everyone would find a Dunbar-sized community in which they can distinguish ...
Linkpost: Arena’s New Opening Hand Rule Has Huge Implications For How We Play the Game
by Zvi — 2018-11-01T12:30:00.810Z

This is a link-post for my latest Magic article, posted on Channel-Fireball: Arena’s New Opening Hand Rule Has Huge Implications For How We Play the Game.

Posting here is a more reliable way to engage with me than commenting there, but they have a robust comment section of other Magic players.
Goodhart's Law and Genies
by thomascolthurst — 2018-11-01T01:38:41.933Z

[Epistemic status: Written in 2010. Possibly of historic interest only.]

Goodhart's Law states that "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

"Control" here means "policy", such as conducted by a government or central bank. To give just one classic example, the Federal Reserve might observe a strong correlation between inflation and unemployment rates which breaks down once the Fed tries to manipulate the inflation rate to reduce unemployment. [1]

The "any" in Goodhart's Law overstates the case -- I regularly use the observed statistical regularity between eating and feeling less hungry for control purposes, and as far as I can tell, this regularity hasn't collapsed yet -- but it isn't hard to see why it might often be the case. If A and B are statistically correlated, there are a number of different causal patterns that could give this correlation: A causes B, B causes A, C causes A and B, etc. Only one of the patterns "works" for the purposes of changing B through A.

The real world situations which Goodhart's Law attempts to explain are even more complicated still. For the inflation/unemployment example, one plausible explanation is that before the Fed started messing with them, inflation rates carried information about macroeconomic conditions, and businesses used this encoded information when making hiring decisions. Post-Fed manipulation, inflation rates no longer contain or correlate with the macro info, so businesses stop using them, and you end up with 70's style stagflation.

A variant of Goodhart's Law makes designing a super-powerful entity harder than it might first appear. (I know, I know, you have this problem all the time.) Specifically:

The actions of any super-powerful constructed entity that identifies important concepts through observed statistical regularities will tend to break the connection between those concepts and the observed statistical regularities.

For example, let's say you want some mangos; this naturally leads you to make a robot to get you some mangos. Now, you could have all sorts of trouble with your robot. It might break down before you get your mangos, it might steal the mangos from a store rather than taking them from your kitchen, it might bring you an or...