trentmkelly/LessWrong-43k huggingface dataset:
Model Mis-specification and Inverse Reinforcement Learning
by Owain_Evans — 2018-11-09T15:33:02.630Z

Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin's note: While I motivated the last post with an example of using a specific model for human biases, in this post (original here), Jacob Steinhardt and Owain Evans point out that model mis-specification can arise in other parts of inverse reinforcement learning as well. The arguments here consider some more practical concerns (for example, the worries about getting only short-term data for each human would not be a problem if you had the entire human policy).

In my previous post, “Latent Variables and Model Mis-specification”, I argued that while machine learning is good at optimizing accuracy on observed signals, it has less to say about correctly inferring the values for unobserved variables in a model. In this post I’d like to focus in on a specific context for this: inverse reinforcement learning (Ng et al. 2000, Abbeel et al. 2004, Ziebart et al. 2008, Ho et al 2016), where one observes the actions of an agent and wants to infer the preferences and beliefs that led to those actions. For this post, I am pleased to be joined by Owain Evans, who is an active researcher in this area and has co-authored an online book about building models of agents (see here in particular for a tutorial on inverse reinforcement learning and inverse planning).

Owain and I are particularly interested in inverse reinforcement learning (IRL) because it has been proposed (most notably by Stuart Russell) as a method for learning human values in the context of AI safety; among other things, this would eventually involve learning and correctly implementing human values by artificial agents that are much more powerful, and act with much broader scope, than any humans alive today. While we think that overall IRL is a promising route to consider, we believe that there are also a number of non-obvious pitfalls related to performing IRL with a mis-specified model. The role of IRL in AI safety is to infer human values, which are represented by a reward function or utility function. But crucially, human values (or human reward functions) are never directly observed.

Below, we elaborate on these issues. We hope that by being more aware of these issues, researchers working on inverse reinforcement learning can anticipate and address the resulting failure modes. In addition, we think that considering issues caused by model...
Prediction-Augmented Evaluation Systems
by ozziegooen — 2018-11-09T10:55:36.181Z

[Note: I made a short video of myself explaining this document here.]

It's common for groups of people to want to evaluate specific things. Here are a few examples I'm interested in:

The expected value of projects or actions within projects

Research papers, on specific rubrics

Quantitative risk estimates

Important actions that may get carried out by artificial intelligences

I think predictions could be useful in scaling and amplifying such evaluation processes. Humans and later AIs could predict intensive evaluation results. There has been previous discussion on related topics, but I thought it would be valuable to consider a specific model here called "prediction-augmented evaluation processes." This is a high-level concept that could be used to help frame future discussion.

Desiderata:

We can call a systematized process that produces evaluations an "evaluation process." Let's begin with a few generic desiderata of these.

High Accuracy / "Evaluating the right thing"

Evaluations should aim at estimating the thing actually cared about as well as possible. In their limit according to some metric of effort, they should approximate ideal knowledge on the thing cared about.

High Precision / "Evaluating the chosen thing correctly"

Evaluations should have low amounts of uncertainty and be very consistent. If the precision is generally less than what naive readers would guess, then these evaluations wouldn't be very useful.

Low Total Cost

Specific evaluations can be costly, but the total cost across evaluations should be low.

I think that the use of predictions could allow us to well fulfill these criterions. It could help decouple evaluations from their scaling, allowing for independent optimization of the first two. The cost should be low relative to that of scaling evaluators in other obvious ways.

Prediction-Augmentation Example

Before getting formal with terminology, I think a specific example would be helpful.

Say Samantha scores research papers for quality on a scale from 1-10. She's great at it, she has a very thorough and lengthy reviewing procedure, and many others trust her reviews. Unfortunately, there's only one Samantha, and there are tons of research papers.

One way to scale Samantha's abilities would be to use a prediction aggregation system. A collection of other people would predict Samantha's scores before she rates them. Predictions wo

...
Update the best textbooks on every subject list
by ryan_b — 2018-11-08T20:54:35.300Z

I occasionally refer back to lukeprog's Best Textbooks on Every Subject post. I thought it might be a good idea to direct people back to it in the hopes of updating the list, for the following reasons:

Old lesswrong is fully integrated now, so we can do it from this site.
It hasn't been significantly updated in 8 years; since then it seems like the community has both diversified and increased in size, so hopefully we can both broaden and deepen the list.
In the interim there have been several rounds of people who have done various levels of MIRI's research guide, and it seems like there is richer engagement with the fields surrounding rationality.
A lot of textbooks get published in 8 years; new ones may be improvements over the old, or we may have gained textbooks for fields which lacked them previously.
I would be particularly interested in anything which accounts for the replication crisis, especially with respect to important fields like behavioral economics.

At ChristianKI's suggestion:

Here are the rules:

1. Post the title of your favorite textbook on a given subject.

2. You must have read at least two other textbooks on that same subject.

3. You must briefly name the other books you've read on the subject and explain why you think your chosen textbook is superior to them.
Multi-Agent Overoptimization, and Embedded Agent World Models
by Davidmanheim — 2018-11-08T20:33:00.499Z

I think this expands on the points being made in the recently completed Garrabrant / Demski Embedded Agency sequence. It also serves to connect a paper I wrote recently that discusses mostly non-AI risks from multiple agents that expands on the work done last year of Goodhart's Law back to the deeper questions that MIRI is considering. Lastly, it tries to point out a bit of how all of this connects to some of the other streams of AI safety research.

Juggling Models

We don't know how to make agents contain a complete world model that includes themselves. That's a hard enough problem, but the problem could get much harder - and in some applications it already has. When multiple agents need to have world models, the discrepancy between the model and reality can have some nasty feedback effects that relate to Goodhart's law, which I am now referring to more generally as overoptimization failures.

In my recent paper, I discuss the problem when multiple agents interact, using poker as a motivating example. Each poker-playing agent needs to have a (simplified) model of the game in order to play (somewhat) optimally. Reasonable heuristics and Machine Learning already achieve super-human performance in "heads-up" (2-player) poker. But the general case of multi-player poker is a huge game, so the game gets simplified.

This is exactly the case where we can transition just a little bit from the world of easy decision theory, which Abram and Scott point out allows modeling "the agent and the environment as separate units which interact over time through clearly defined i/o channels," to the world of not embedded agents, but interacting agents. This moves just a little bit in the direction of "we don't know how to do this."

This partial transition happens because the agent must have some model of the decision process of the other players in order to play strategically. In that model, agents need to represent what those players will do not only in reaction to the cards, but in reaction to the bets the agent places. To do this optimally, they need a model of the other player's (perhaps implicit) model of the agent. And building models of other player's models seems very closely related to work like Andrew Critch's paper on Lob's Theorem and Cooperation.

That explains why I claim that building models of c...
Embedded Curiosities
by Scott Garrabrant — 2018-11-08T14:19:32.546Z

A final word on curiosity, and intellectual puzzles:

I described an embedded agent, Emmy, and said that I don't understand how she evaluates her options, models the world, models herself, or decomposes and solves problems.

In the past, when researchers have talked about motivations for working on problems like these, they’ve generally focused on the motivation from AI risk. AI researchers want to build machines that can solve problems in the general-purpose fashion of a human, and dualism is not a realistic framework for thinking about such systems. In particular, it's an approximation that's especially prone to breaking down as AI systems get smarter. When people figure out how to build general AI systems, we want those researchers to be in a better position to understand their systems, analyze their internal properties, and be confident in their future behavior.

This is the motivation for most researchers today who are working on things like updateless decision theory and subsystem alignment. We care about basic conceptual puzzles which we think we need to figure out in order to achieve confidence in future AI systems, and not have to rely quite so much on brute-force search or trial and error.

But the arguments for why we may or may not need particular conceptual insights in AI are pretty long. I haven't tried to wade into the details of that debate here. Instead, I've been discussing a particular set of research directions as an intellectual puzzle, and not as an instrumental strategy.

One downside of discussing these problems as instrumental strategies is that it can lead to some misunderstandings about why we think this kind of work is so important. With the “instrumental strategies” lens, it’s tempting to draw a direct line from a given research problem to a given safety concern. But it’s not that I’m imagining real-world embedded systems being “too Bayesian” and this somehow causing problems, if we don’t figure out what’s wrong with current models of rational agency. It’s certainly not that I’m imagining future AI systems being written in second-order logic! In most cases, I’m not trying at all to draw direct lines between research problems and specific AI failure modes.

What I’m instead thinking about is this: We sure do seem to be working with the wrong basic concepts today when we try to think about what agency is, as seen by th...
On first looking into Russell's History
by Richard_Ngo — 2018-11-08T11:20:00.935Z
Link: http://thinkingcomplete.blogspot.com/2018/08/on-first-looking-into-russells-history.html

Notes from Bertrand Russell's History of Western Philosophy, in particular the sections on ancient philosophy and theology.

Western civilisation began in the city-states of Greece - and there wasn't any comparable intellectual and artistic flourishing until the city-states of Italy during the Renaissance. Russell argues that this isn't a coincidence: that the intellectual climate of a time is bound up in the political and historical circumstances.
He also draws several dichotomies: between optimistic and pessimistic times; between individualistic and grouped times; also systematic and piecemeal; also subjective and objective. Some of these are useful perspectives, although they're the sort of categories we should be careful not to over-generalise.
Russell holds that the first philosophers created scientific hypotheses which were, if not exactly testable, at least empirically meaningful. These included Thales (around 585 BC; he held that everything was made out of water) and his Milesian School, Anaximander (everything was made of one elemental substance), Anaximenes (air), Heraclitus (fire and change), Parmenides (no change), Empedocles (4 elements), Anaxagoras (infinite divisibility) and eventually Democritus (atoms), who was perhaps the most rigorous, though his correctness was still essentially accidental. These were also, notably, all reductionist doctrines, in that they were concerned with analysing objects in terms of their simpler components; and non-teleological (although Anaximander incorporated a notion of cosmic "justice").
We must also include Pythagoras as the founder of mathematics (in the sense of demonstrative deductive arguments), whose influence is a source of the belief in "eternal and exact truth" in philosophy. He forwarded a view of philosophers as contemplative gentleman aristocrats, and foreshadowed Plato's forms.
Next, the Sophists, led by Protagoras, who claimed the skeptical hypothesis that "[each] man is the measure of all things."
After this, we encounter Socrates, Pluto and Aristotle, the three greatest names in ancient philosophy but also (along with Protagoras) heralds of a turn towards human-centred philosophy. Socrates, insofar as we can separate him from Plato, was largely concerned with ethical questions, which he approached using the Socratic method of conceptual analysis,

...
Is Copenhagen or Many Worlds true? An experiment. What? Yes.
by Jonathanm — 2018-11-08T10:10:20.396Z
Link: https://arxiv.org/abs/1811.02983v1

If this paper is right, an experiment should be possible.

Building it also doesn't seem that hard to do technically, so I think it could be interesting to watch in the coming months/years.

If Many Worlds dies, we would seem to have a perpetual motion machine of the second kind. But if the second law of thermodynamics prevails, Copenhagen should provably be wrong.

All, if this paper is right. Does somebody find a mistake?
The new Effective Altruism forum just launched
by habryka — 2018-11-08T01:59:01.502Z

CEA just launched the new EA forum, now located at forum.effectivealtruism.org. It's based on the LessWrong codebase, and we've collaborated with them on getting everything up and running and making the codebase adaptable for them, as well as helping fix up various bugs and problems that they ran into while setting everything up.

I expect many LW readers will find some of the content there interesting, and I am also looking forward to how the changes they made to the site will work out, as that will inform how we develop LessWrong in the future.

Their opening post also had some interesting things to say about intellectual progress and the purpose of the platform, which also applies to LessWrong in at least some aspects:

[Quotes section from here]

Making progress as a community

Much intellectual progress is done as part of a community: even when an individual has an important new insight, it tends to take many people to bring the idea to maturity. Academic science is perhaps the most obvious example of this.

Intellectual progress is not just something done by professional philosophers. It’s something that everyone in the community can contribute to.

Just as you might donate some of your income without working full-time for a charity, you can contribute intellectually without becoming a full-time researcher. For example, you could aggregate data on the number of invertebrates in the world, tweak GiveWell’s cost-effectiveness model, or analyze the history of an interesting social movement. And you can also help by writing up explanations or summaries of others’ work. Whilst you do so, you’ll be sharpening your thinking, understanding others’ ideas better, and learning about important topics.

This is why open forums have historically been places where key ideas behind effective altruism have been hammered out -- for instance, Felicifia, LessWrong, SL4, and many on the EA Forum itself.

To make intellectual progress, the community needs the right infrastructure, amongst other things:

A place to share ideas (e.g. journals)
A way of getting feedback on intellectual work (e.g. discussion groups, online forums)
An easy way to search for existing work (e.g. literature reviews)
Shared norms of discussion, and standards for work (e.g. karma voting)
Common knowledge of core ideas, so that ideas can be built upon rather than constantly rediscovered (e.g. textbooks)
A place

...
What are Universal Inductors, Again?
by Diffractor — 2018-11-07T22:32:57.364Z

[Attention Conservation Notice: This just recaps an old result and patches a hole in it, it doesn't contain substantially new ideas.]

Universal Inductors can be thought of as logical inductors over bitstrings, which are notable because they can act as a logical inductor over any theory you want (PA, ZFC, 2nd order arithmetic), by fixing some efficiently computable function from bits to sentences in the language of your theory, conditioning the universal inductor (which is a probability distribution over infinite bitstrings) on the bits which correspond to proven theorems, and reading out the probability of other bits/sentences from the resulting conditional distribution. This trick works by Theorem 4.7.2 in the logical induction paper, "Closure Under Conditioning", which shows that conditioning the probabilities of a logical inductor on a sequence of non-inconsistent sentences, yields a logical inductor over a deductive process where all the conditionals are true.

Also, since they are a probability distribution over infinite bitstrings, it makes sense to think about them as having a probability measure over worlds (assignments of statements to true or false, for all statements) at all finite stages, instead of just in the limit.

However, in Scott's old post about universal inductors, their construction was never described.

Note that a Universal Inductor corresponds to a Logical Inductor, but the associated Logical Inductor will not have finite support, and so will be different from the one constructed in the paper. Never the less, Universal Inductors can be shown to exist using a similar construction.

This post will give that construction. To begin with, a Universal Inductor must fulfill the following two properties:

First, it must be a distribution over infinite bitstrings, such that Pn(σ) is computable for all n and all σ which are finite bitstrings. This gives the probability that an infinite bitstring starts with σ as a prefix.

Second, the theory is one where the nth atomic statement is of the form "the nth bit in this infinite bitstring is a 1", and Pn induces a function from these statements to probabilities, and the sequence of probability assignments must form a logical inductor over the empty deductive process. (ie, the inductor never sees any evidence of the form "this bit is a 1", it just runs forever without get...
Burnout: What it is and how to Treat it.
by Elizabeth — 2018-11-07T22:02:23.649Z
Link: https://forum.effectivealtruism.org/posts/NDszJWMsdLCB4MNoy/burnout-what-is-it-and-how-to-treat-it

I reviewed the scientific literature on burnout to create a better definition and treatment plan.

Tl;dr

Social support == Good.

Sleep == Good.

Ambiguity == Bad.

Vacations == Meh.

I expect a lot of value in the comments and want to keep them all in one place, so comments will be disabled here.
Bayes Questions
by Bucky — 2018-11-07T16:54:38.800Z

For the first time ever I’ve had reason to use formal Bayesian statistics in my work. I feel this is a cause for streamers and confetti.

However, I’ve got a bit stuck on analysing my confidence levels and I thought what better place than lesswrong to check that I’m making sense. I’m not sure this is strictly what lesswrong is for but the site says that this is my own personal blog so I guess its ok?! I can always be down-voted to hell if not!

***

I’m trying to calculate estimated life before failure of a particular component.

We’ve done a number of tests and the results show a larger than expected variance with some components not failing even after an extended lifetime. I’m trying to analyse the results to see which probabilistic failure distribution best suits the available data. I have three different distributions (Weibull, Log-normal & Birnbaum-Saunders) each of which has a shape parameter and a scale parameter.

For each distribution I created a grid which samples the possible values of these parameters. I’ve given the parameters a log uniform prior by giving each sampled parameter pair a uniform prior but sampling the parameters geometrically (i.e. each sampled value of the parameter is a fixed multiple of the previous value). I’ve tried other priors and the results seem fairly robust over choice of prior.

For components which have failed, P(E|H) is the probability density function at the number of hours before failure.

For components which get to a certain age and do not fail, P(E|H) is 1 – the cumulative probability function at this number of hours.

This is implemented on a spreadsheet with a tab for each test result. It updates the prior probability into a posterior probability and then uses this as the prior for the next tab. The result is normalised to give total probability of 1.

Initially I calculate the expected life of the worst component in 1,000,000. For this, I just use the inverse cumulative probability function with p=0.000001 and calculate this for all of the potential probability distributions.

The results of this calculation are multiplied by the final probabilities of each distribution being the correct one. Then I sum this over the entire hypothesis space to give the expected life of the worst component in a million.

So my first question – is all of the above sound or have I made some silly mistake in my logic?

***

The part that I’m less co...