trentmkelly/LessWrong-43k huggingface dataset:
Alignment Newsletter #33
by Rohin Shah — 2018-11-19T17:20:03.463Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through the database of all summaries.

One correction to last week's newsletter: the title Is Robustness at the Cost of Accuracy should have been Is Robustness the Cost of Accuracy.

Highlights

Reward learning from human preferences and demonstrations in Atari (Borja Ibarz et al): We have had lots of work on learning from preferences, demonstrations, proxy rewards, natural language, rankings etc. However, most such work focuses on one of these modes of learning, sometimes combined with an explicit reward function. This work learns to play Atari games using both preference and demonstration information. They start out with a set of expert demonstrations which are used to initialize a policy using behavioral cloning. They also use the demonstrations to train a reward model using the DQfD algorithm. They then continue training the reward and policy simultaneously, where the policy is trained on rewards from the reward model, while the reward model is trained using preference information (collected and used in the same way as Deep RL from Human Preferences) and the expert demonstrations. They then present a lot of experimental results. The main thing I got out of the experiments is that when demonstrations are good (near optimal), they convey a lot of information about how to perform the task, leading to high reward, but when they are not good, they will actively hurt performance, since the algorithm assumes that the demonstrations are high quality and the demonstrations "override" the more accurate information collected via preferences. They also show results on efficiency, the quality of the reward model, and the reward hacking that can occur if you don't continue training the reward model alongside the policy.

Rohin's opinion: I'm excited to see work that combines information from multiple sources! In general with multiple sources you have the problem of figuring out what to do when the sources of information conflict, and this is no exception. Their approach tends to prioritize demonstrations over preferences when the two conflict, and so in cases where the preferences are better (as in Enduro) their approach performs poorly. I'm somewhat surprised that they prioritize demos over preferences, since it seems humans would be more reliable at prov...
Letting Others Be Vulnerable
by  — 2018-11-19T02:59:21.423Z

[Because people are predisposed to give others good impressions, when our relationships deepen, it’s more likely that we’ll have to revise our initial “perfect” models of the other person. For the person doing the updating, it can internally feel uncomfortable. I outline this dynamic and argue why it’s important to push through nonetheless.]

Recently, I drew a comic about the importance of being accepting when someone opens up to you. I think I did a passable job of condensing the main points into the text, but I figured that a more in-depth explanation was probably worth a blog post in itself. Plus, it’s been a while since the last blog post, and I’ve been doing a very poor job sticking to my supposed weekly release schedule.

Here’s the comic, if you haven’t seen it:

Before we begin, a quick clarification: I am agnostic about whether or not the relationships I describe below are of a romantic nature or not. I think that the process of “becoming closer to someone” is roughly the same, no matter the relationship type.

In the above comic, it’s not exactly clear how people are supposed to be “interacting”. To make things clearer, below is an illustration simplified model I’ll be using throughout the essay. (I stole the idea of representing people as 2-D shapes from Kevin Simler’s fantastic essay on Personhood, which I’d also recommend checking out.)

This is a person:

They have a lot of things inside of them, and some of them get shown to other people:

On the outside, on their surface edges, they have nice, smooth, grooves on the outside. They are the pieces that make connecting with other people easy.

On the inside, we have lots of pointier pieces that we keep hidden, the pieces which might cause disagreement. If we were to let them out, they might grate against other people.

Our external surface represents the facets of ourselves that we show, be it the words we speak, the reactions we show, and the body language we display. Our interactions can be thought to involve our edges bumping into each other. So we’re choosy about what gets shown and what gets kept hidden.

Sometimes, this choosiness is explicit, like selecting which jokes to tell to signal you’re part of some in-group (EX: “If we’re ordering pizza, make sure to get two boxes!”). Other times, it’s implicit, like the pauses in our speech and the way we behave (EX: interrupting less when the other speaker seems im...
Clickbait might not be destroying our general Intelligence
by Donald Hobson — 2018-11-19T00:13:12.674Z

Epistemic status This post is a "plausible conjecture" alternative to Eliezer's take on clickbait. My arrogant epistimology thinks it's substantially closer to the truth than Eliezer's version.

https://www.lesswrong.com/posts/YicoiQurNBxSp7a65/is-clickbait-destroying-our-general-intelligence

In the 1950's there were a similar number (order of magnitude) of humans, and they did a similar amount of socializing as today. This would suggest that a similar quantity of memetic optimization was going on then. Any universally popular meme would have risen to popularity. The difference between then and today was that then people usually only communicated with people geographically local to them. With a small number of newspapers being widely read. Whereas today, people communicate with like minded individuals around the world regularly.

Suppose that different humans have different selection criteria when deciding to share a meme. In 1950, a meme had to appeal to a broad section of society to be spread. Not 100%, more like 25%, the small number of different newspapers, and small numbers of local people to communicate with still let memes specialize along a few socioeconomic lines. Eg Catholic memes vs Protestant memes in Northern Ireland, or Liberal vs Conservative memes in many places. Each newspaper had a side, and many of your neighbors were on your side.

Nowadays, memes can specialize to focus onto tiny subsets of the population. Given that a tiny fraction of the population think in a particular and unusual way, they can gather together online, and share memes optimized exclusively to them. This produces many internet subcultures. Within each subculture, the selection pressure isn't that strong, there aren't that many model railway buffs or code golfers or ... But the memes are optimized to a particular way of thinking, not the combination of several.

In some circumstances, Schelling points create an averaging effect in the 1950 case. Suppose an issue, say cats vs dogs, is sufficiently minor that newspapers are not split into a pro cat paper, and a pro dog paper. Then if a newspaper says anything excessively pro cat, the dog fans will call them out on it, and possibly stop reading the paper. Likewise if the paper is pro dog. So the paper finds a Schelling point, where neither side are upset enough to cause a real fuss. Note...
[Link] "They go together: Freedom, Prosperity, and Big Government"
by CronoDAS — 2018-11-18T16:51:00.579Z

Link to article

Our statistical investigations lead to two substantive conclusions:

First, the data appear to support notion that economic freedom makes a positive contribution to personal freedom and prosperity. That holds true whether we measure prosperity in a narrowly economic sense, as GDP per capita, or in a broader sense, using noneconomic indicators of education, health, and personal safety.

Second, the data do not support the notion that a larger government is necessarily detrimental to either freedom or prosperity. On the contrary, countries with larger government sectors tend to have more personal freedom and higher indicators of education, health, and personal safety.
Collaboration-by-Design versus Emergent Collaboration
by Davidmanheim — 2018-11-18T07:22:16.340Z

Introduction

This seems to be a non-trivial problem for even current narrow AI, which is much more problematic for strong NAI, which I haven't seen called out or named explicitly. I provide a quick literature review to explain why I think it's ignored in classic multi-agent system design. (But I might be corrected)

It is unclear to me whether we can expect even introspective GAI to "just solve it" by noticing that it is a problem and working to fix it, given that people often don't seem to manage it.

The problem

One challenge for safe AI is the intrinsic difficulty of coordination problems. This includes coordination with humans, coordination with other AI systems, and potentially self-coordination when AI uses multiple agents. Unfortunately, the typical system design intends to maximize some fitness function, not to coordinate in order to allow mutually beneficial interaction.

There is extensive literature on multi-agent coordination for task-based delegation and cooperation, dating back to at least the 1980 Contract Net Interaction Protocol, which allows autonomous agents to specify markets for interaction. This is useful, but doesn't avoid any of the problems with market failures and inadequate equillibria. (In fact, it probably induces such failures, since individual contracts are the atomic unit or interaction.) Extensive follow-up work on distributed consensus problems assumes that all agents are built to achieve consensus. This may be important for AI coordination, but requires clearly defined communication channels and well-understood domains. Work on Collaborative Intelligence is also intended to allow collaboration, but it is unclear that there is substantive ongoing work in that area. Multiscale decision theory attempts to build multi-scale models for decision making, but is not tied explicitly to multiple agents.

What most of the literature shares is an assumption that agents will be designed for cooperation and collaboration. Inducing collaboration in agents not explicitly designed for that task is a very different problem, as is finding coordinated goals that can be achieved.

"Solutions"?

The obvious solution is to expect multi-agent systems to have agents with models of other agents that are sophisticated enough to build strategies that allow collaboration. In situations where multiple equilibria exist, moving f...
Diagonalization Fixed Point Exercises
by Scott Garrabrant — 2018-11-18T00:31:19.683Z

This is the second of three sets of fixed point exercises. The first post in this sequence is here, giving context.

Recall Cantor’s diagonal argument for the uncountability of the real numbers. Apply the same technique to convince yourself than for any set S, the cardinality of S is less than the cardinality of the power set P(S) (i.e. there is no surjection from S to P(S)).

Suppose that a nonempty set T has a function f from T to T which lacks fixed points (i.e. f(x)≠x for all x∈T). Convince yourself that there is no surjection from S to S→T, for any nonempty S. (We will write the set of functions from A to B either as A→B or BA; these are the same.)

For nonempty S and T, suppose you are given g:S→TS a surjective function from the set S to the set of functions from S to T, and let f be a function from T to itself. The previous result implies that there exists an x in T such that f(x)=x. Can you use your proof to describe x in terms of f and g?

Given sets A and B, let Comp(A,B) denote the space of total computable functions from A to B. We say that a function from C to Comp(A,B) is computable if and only if the corresponding function f′:C×A→B (given by f′(c,a)=f(c)(a)) is computable. Show that there is no surjective computable function from the set S of all strings to Comp(S,{T,F}).

Show that the previous result implies that there is no computable function halt(x,y) from S×S→{T,F} which outputs T if and only if the first input is a code for a Turing machine that halts when given the second input.

Given topological spaces A and B, let Cont(A,B) be the space with the set of continuous functions from A to B as its underlying set, and with topology such that f:C→Cont(A,B) is continuous if and only if the corresponding function f′:C×A→B (given by f′(c,a)=f(c)(a)) is continuous, assuming such a space exists. Convince yourself that there is no space X which continuously surjects onto Cont(X,S), where S is the circle.

In your preferred programming language, write a quine, that is, a program whose output is a string equal to its own source code.

Write a program that defines a function f taking a string as input, and produces its output by applying f to its source code. For example, if f reverses the given string, then the program should outputs its source code backwards.

Given two sets A and B of sentences, let Syn(A,B) be the set of all functions f

...
Ia! Ia! Extradimensional Cephalopod Nafl'fhtagn!
by ExCeph — 2018-11-17T23:00:05.983Z

Hello, I'm ExCeph (short for Extradimensional Cephalopod). I've been mostly lurking here for the past decade and occasionally commenting within the past year or so. However, I never got around to actually introducing myself until now. It's good to meet you.

I was advised by a friend to actually write up a short bio, so here’s a bit about me.

*starts puffing a bubble pipe*

I grew up on science fiction and fantasy, plowing through satirical books such as the Discworld series by Terry Pratchett and the MythAdventures series by Robert Asprin, which inspired me to approach problems with a clear and critical mind.

One of my high school friends introduced me to both Less Wrong and Harry Potter and the Methods of Rationality, which resonated with me. I read

HPMOR

up through its then most recent update and moved on to other rational fiction, like

Luminosity

and

Elcenia

. I also acquired an addiction to taste for reading

TVTropes

, which has

Enhanced My Life

. (But ask your doctor if TVTropes is right for you).

Reading through some Less Wrong and

Slate Star Codex

articles about the inadequacy of existing institutions confirmed my childhood suspicions that humans didn’t really know what they were doing with the world. I became extremely frustrated that fixing these institutions was not a public priority, as if no one was aware of what was wrong or how to approach the situation.

It was the summer before my senior year of high school when I came up with the first version of my insanely ambitious plan to change the world, which has been my main project ever since. As part of the plan, I researched and tested different concepts and tools to improve my productivity and my ability to effect positive change in the world, with sporadic progress.

During my first job out of college, I reached out to find other people who wanted to change the world, and that led me to the local effective altruism community. As a result, I spent some time volunteering with an effective altruist organization. Eventually, through both the job and the EA organization, I discovered I was still missing some fundamental attributes that nobody responsible for my education had noticed or figured out how to instill in me.

Since then I've spent the past year and a half refining and practicing the tools and attributes I'd discovered, and am now ready to participate in the rati...
An unaligned benchmark
by paulfchristiano — 2018-11-17T15:51:03.448Z

My goal is to design AI systems that are aligned with human interests and competitive with unaligned AI.

I find it useful to have a particular AI algorithm in mind. Then I can think about how that algorithm could cause trouble, and try to find a safer variant.

I think of the possibly-unaligned AIs as a benchmark: it’s what AI alignment researchers need to compete with. The further we fall short of the benchmark, the stronger the competitive pressures will be for everyone to give up on aligned AI and take their chances.

I have a few standard benchmarks I keep in mind. This post describes one of those benchmarks. It also tries to lay out clearly why I think that benchmark is unsafe, and explains how I think my current research could make a safe version.

I. Model-based RL with MCTS

We train three systems in parallel:

A generative model to sample sequences of observations, conditioned on sequences of actions.
A reward function that takes as input a sequence of actions and predicted observations and produces a reward.
A policy and value function which take as input a sequence of observations and produce the next action and an estimate of the future return.

We train the policy and value function using (roughly) the AlphaZero algorithm: Use MCTS to improve the current policy. Update the policy at the root to predict the best move found by MCTS, update the value to predict its predicted value. Use the generative model to sample environment transitions and the reward function (with a small discount rate) to score them.

We train an autoregressive generative model, to maximize the log probability assigned to the actual sequence of actions and observations produced by the AI (with each observation conditioned on the past actions). This isn’t actually a good way to train the generative model, but it’s not really central to the discussion.

We train the reward function by showing humans sequences of actions and predicted observations, asking them to assign scores, then predicting those scores with supervised learning. We show humans the sequences of actions that look most promising to the system.

There are plenty of details you’d need in order to make this work well, but that’s the basic idea. When applied with very powerful networks, it’s plausible that this system would be able to decisively outcompete humans. It would be capable performing a large intelligent search over long seque...
On Rigorous Error Handling
by Martin Sustrik — 2018-11-17T09:20:00.753Z

In my long career as a programmer I've seen a lot of projects with a lot of error handling practices and, sadly, almost none that actually worked.

We are facing facing few high-level problems here:

For many applications error handling is not critical.

Error handling code is rarely executed. Consequently, it looks like it works until it doesn't.

Programmers want to implement new features. Writing error handling is just an annoyance that slows them down.

Who cares about error handling?

For many applications writing rigorous error handling code doesn't pay off.

As long as you have some semi-decent way to deal with errors transparently (e.g. exceptions) you are fine. If there's a problem, throw an exception. On the top level catch all the exceptions and open a dialog box with the error message (in frontend applications) or write it to the log (on the server).

The above, of course, means that you hope that, after the exception is processed, the application will we left in a consistent, functional state. Once again, you HOPE. You do not KNOW. You do not know because, when writing the application, you haven't thought about error handling at all.

Now, don't get me wrong. I am not saying that you should never do this. It often makes sense from economic perspective. Writing rigorous error handling is expensive. If the worst thing that can happen is that a frontend application crashes once in a long while making user curse and restart it, then it's totally not worth doing.

However, there is also a different kind of application. There are medical application where the patient dies if they misbehave. There are space probes that crash into plantes, taking millions of dollars of investment as well as scientific careers down with them. There are HFT application which can generate bazillion of dollars of loss for every minute of downtime.

In these cases it's definitely worth writing rigorous error handling code.

That much everybody agrees with.

However, it is often not explicitly understood that whenever you are writing a piece of infrastructure (as opposed to a client-facing application) you are always in the latter category.

A piece of infrastructure can be used in a medical application, in a space probe, in a high-frequency trading system, but unless it's a commercial product and you take care not to sell it to anyone who looks too important to fail, then you have no w

...
Act of Charity
by jessicata — 2018-11-17T05:19:20.786Z

(Cross-posted from my blog)

The stories and information posted here are artistic works of fiction and falsehood. Only a fool would take anything posted here as fact.

—Anonymous

Act I.

Carl walked through the downtown. He came across a charity stall. The charity worker at the stall called out, "Food for the Africans. Helps with local autonomy and environmental sustainability. Have a heart and help them out." Carl glanced at the stall's poster. Along with pictures of emaciated children, it displayed infographics about how global warming would cause problems for African communities' food production, and numbers about how easy it is to help out with money. But something caught Carl's eye. In the top left, in bold font, the poster read, "IT IS ALL AN ACT. ASK FOR DETAILS."

Carl: "It's all an act, huh? What do you mean?"

Worker: "All of it. This charity stall. The information on the poster. The charity itself. All the other charities like us. The whole Western idea of charity, really."

Carl: "Care to clarify?"

Worker: "Sure. This poster contains some correct information. But a lot of it is presented in a misleading fashion, and a lot of it is just lies. We designed the poster this way because it fits with people's idea is of a good charity they should give money to. It's a prop in the act."

Carl: "Wait, the stuff about global warming and food production is a lie?"

Worker: "No, that part is actually true. But in context we're presenting it as some kind of imminent crisis that requires an immediate infusion of resources, when really it's a very long-term problem that will require gradual adjustment of agricultural techniques, locations, and policies."

Carl: "Okay, that doesn't actually sound like more of a lie than most charities tell."

Worker: "Exactly! It's all an act."

Carl: "So why don't you tell the truth anyway?"

Worker: "Like I said before, we're trying to fit with people's idea of what a charity they should give money to looks like. More to the point, we want them to feel compelled to give us money. And they are compelled by some acts, but not by others. The idea of an immediate food crisis creates more moral and social pressure towards immediate action, than the idea that there will be long-term agricultural problems that require adjustments.

Carl: "That sounds...kind of scammy?"

Worker: "Yes, you're starting to get it! The act is about viole

...