trentmkelly/LessWrong-43k huggingface dataset:
Bounded Oracle Induction
by Diffractor — 2018-11-28T08:11:28.183Z

Introduction:

Because logical induction relies on the Brouwer fixed-point theorem, and reflective oracles rely on the Kakutani fixed-point theorem which Brouwer is a special case of, it's possible that logical induction could have been derived for the first time from reflective oracles. Attempting to do this in the most obvious way by having traders output a circuit that takes a binary-search approximation of the market as input doesn't produce any insights in particular. However, by attempting to redevelop the logical induction algorithm from scratch with the aid of a bounded reflective oracle, we arrive at a new way of looking at logical induction, with the following interesting features.

1: It collapses the distinction between the algorithm outputting the trading circuit, and the trading circuit itself.

2: All trades can be naturally interpreted as probability measures over bitstrings, with the reward given by a simple betting game, instead of shares that pay off if some boolean combination is true. However, the betting game doesn't incentivize true belief reporting, just moving the probability distribution of the market in the right direction. This turns out to be isomorphic to the original formulation of the value of a trade, subject to one extra restriction on worst-case trade value.

3: The market prices are also a probability measure over bitstrings/worlds, exactly like a universal inductor.

4: It does the reflective-solomonoff thing of "draw a turing machine with probability 2−K-complexity, use it to predict future bits"

First, notation will be established, and the algorithm will be discussed. Then the connection between OI-trader scoring and LI-trader scoring will be established, and questions of how many of the nice properties of LI carry over will be discussed. We will finish with basic discussion of what this result means, and then there will be an appendix which contains the definitions of auxiliary algorithms, and the next post will contain the proof that the algorithm is an Oracle Inductor.

As a quick refresher, a bounded reflective oracle takes as input a query of the form (A,p), and returns 1 if A has bounded runtime, and only makes oracle calls to algorithms with the same runtime bound, and outputs 1 with probability greater than p, and 0 if A has bounded runtime and only makes oracle calls to algorithms with the same ru...
Double-Dipping in Dunning--Kruger
by isovector — 2018-11-28T03:40:58.206Z

Originally posted at sandymaguire.me.

I have to get you to drop modesty and say to yourself, "Yes, I would like to
do first-class work." Our society frowns on people who set out to do really
good work. You're not supposed to; luck is supposed to descend on you and you
do great things by chance. Well, that's a kind of dumb thing to say. I say,
why shouldn't you set out to do something significant. You don't have to tell
other people, but shouldn't you say to yourself, "Yes, I would like to do
something significant."

Richard Hamming

I want to talk about impostor syndrome today. If you're unfamiliar with the
term, it's this phenomenon where people discount their achievements and feel
like frauds---like at any moment, others will realize that they don't belong.
People with impostor syndrome are convinced that they've somehow pulled the wool
over others' eyes, and have conned their way into their jobs or social circles.
And it's very, very common.

On the other end of the spectrum, there's another phenomenon known as
Dunning--Kruger. This one is widely reported in the media as "the average person
believes they are better than 70% of drivers." They can't all be right---by
definition, 50% of people must be worse than the median driver. This too is very
common; it seems to hold true in most positive-feeling, every-day domains.

While this is indeed supported by Dunning--Kruger's data, the paper itself
focuses on the fact that "the worst performers in a task are often so bad that
they have no idea." For example, the least funny people are so tragically
unfunny that they wouldn't know humor if it bit them. As a result, unfunny
people are entirely blind to their lack of humor:

Participants scoring in the bottom quartile on our humor test not only
overestimated their percentile ranking, but they overestimated it by 46
percentile points.

A less well-known finding of Dunning--Kruger is that the best performers will
systematically underestimate how good they are, by about 15 percentile points.
The proposed reason is that they found the task to be easy, assume others must
have also found it easy, and go from there. In other words, top performers are
so good that they don't notice many challenges.

It's unfortunate that Dunning--Kruger has been popularized as "most people think
they are better than they are." Not only do high-performers already
underestimate themselves, but tho

...
Winter Solstice 2018 Roundup
by Raemon — 2018-11-28T03:09:44.938Z

Many LessWrong communities are holding Winter Solstice holiday events. What exactly that entails varies from community-to-community, but often includes:

A ceremony commemorating humanity's progress over the millenia (often involving speeches and singalong songs)
Meetups and parties coinciding with said ceremony, providing a natural time for many people to travel for a larger-than-usual rationalist gathering.

If you're hosting such an event, this is a good post to comment on linking to whatever event-pages you have set up to collect RSVPs or tickets/donations.

It's helpful reserving a spot ASAP (as well as donating to help pay for the cost of the venue), so that the organizers can plan better. Many events have a pay-what-you-can scheme, with newcomers or people without much spare cash encouraged to donate around $15, and more well-to-do community members can help subsidize the event for others.

Events So Far

US East Coast Megameetup

December 15th – New York City is hosting their annual Solstice + Megameetup. Travelers are welcome all weekend long to stay at a large house being rented for the occasion. The ceremony itself is will be Saturday evening.

RSVP for the Megameetup (either for visiting during-the-day-activities, or reserving space to sleep overnight)
RSVP for the ceremony on Facebook
Pay-What-You-Can on Eventbrite to help support the ceremony

Bay Area

December 9th – The San Francisco Bay Area is holding their solstice event at a planetarium in Oakland. There should be a sufficient number of seats, but seating is finite, so please get an official ticket on Eventbrite to confirm there is still space.

RSVP on Facebook
Pay-what-you-can on Eventbrite

Boston

December 21st – A smaller, more intimate event, Boston wins the "actually celebrating Solstice on Solstice" award.

RSVP on Facebook

Seattle

December 22nd – Honorable mention for "almost on actual Solstice."

RSVP on Facebook
Pay-what-you-can on Eventbrite

I believe there are Solstices in the works in the UK and Germany but do not know many details about them. If you're running one, please comment!
Corrigibility
by paulfchristiano — 2018-11-27T21:50:10.517Z

(Warning: rambling.)

I would like to build AI systems which help me:

Figure out whether I built the right AI and correct any mistakes I made
Remain informed about the AI’s behavior and avoid unpleasant surprises
Make better decisions and clarify my preferences
Acquire resources and remain in effective control of them
Ensure that my AI systems continue to do all of these nice things
…and so on

We say an agent is corrigible (article on Arbital) if it has these properties. I believe this concept was introduced in the context of AI by Eliezer and named by Robert Miles; it has often been discussed in the context of narrow behaviors like respecting an off-switch, but here I am using it in the broadest possible sense.

In this post I claim:

A benign act-based agent will be robustly corrigible if we want it to be.
A sufficiently corrigible agent will tend to become more corrigible and benign over time. Corrigibility marks out a broad basin of attraction towards acceptable outcomes.

As a consequence, we shouldn’t think about alignment as a narrow target which we need to implement exactly and preserve precisely. We’re aiming for a broad basin, and trying to avoid problems that could kick out of that basin.

This view is an important part of my overall optimism about alignment, and an important background assumption in some of my writing.

1. Benign act-based agents can be corrigible

A benign agent optimizes in accordance with our preferences. An act-basedagent considers our short-term preferences, including (amongst others) our preference for the agent to be corrigible.

If on average we are unhappy with the level of corrigibility of a benign act-based agent, then by construction it is mistaken about our short-term preferences.

This kind of corrigibility doesn’t require any special machinery. An act-based agent turns off when the overseer presses the “off” button not because it has received new evidence, or because of delicately balanced incentives. It turns off because that’s what the overseer prefers.

Contrast with the usual futurist perspective

Omohundro’s The Basic AI Drives argues that “almost all systems [will] protect their utility functions from modification,” and Soares, Fallenstein, Yudkowsky, and Armstrong cite as: “almost all [rational] agents are instrumentally motivated to preserve their preferences.” This motivates them to consider modifications to an agent to remove this def...
Summary: Surreal Decisions
by Chris_Leong — 2018-11-27T14:15:07.342Z

This post summarises a paper by Eddy Chen and Daniel Rubio on using Surreal numbers to resolve problems of Infinite Ethics. Future posts will argue that surreals are the correct approach to this problem before extending upon this work. However, this post merely aims to summarise this paper.

Background

The problem of Infinite Paralysis is best described as follows. Suppose that there are infinite people and that they are happy so that there is infinite utility. I then come along and punch 100 people destroying 100 utility. Since there was infinite utility at the start and infinity minus 100 is still infinity, so arguably I’ve done nothing wrong. However, this seems to be a reductio ad absurdum if I’ve ever seen one.

One approach mentioned by Bostrom is to use hyperreals to represent infinite sequences of utility. In particular, he sets the ith index of the hyperreal representing utility to the sum of the first i numbers.

Unfortunately, there is not a unique definition of the hyperreals - they require what’s called a non-principle ultrafilter to be defined in order to determine the ordering. Our choice of this seems essentially arbitrary and therefore hard to justify principally. Additionally, summation requires a preferred location around which to sum, which can be hard to justify philosophically.

Surreals Decisions

Chen and Rubio outline a surreal decision theory by adapting the the Von Neumann-Morgenstern axioms. They then use it to analyse Pascal’s Wager to demonstrate that the validity of the argument depends on particular infinite values assigned in the problem and the various deities that exist.

They note that Expected Utility Theory with standard infinities (cardinal numbers) seems to produce absurd results. In particular, it is indifferent between the following, when most people would prefer them in order

Infinity or something: Infinite utility if heads, 10000 utility if tails
Infinity or nothing: Infinite utility if heads, 0 if tails
Infinity or bust: Infinite utility if heads, -10000 if tails

They provide another example, where they argue that the ordering is obvious, but that this is undefined for Expected Utility Theory:

Biased positive infinity: 9/10 times plus infinity, 1/10 minis infinity
Fair infinity: 50/50 chance of plus infinity or negative infinity
Biased negative infinity: 9/10 times minus infinity, 1/10 plus infinity

They then argue that surreal numbe...
How democracy ends: a review and reevaluation
by Richard_Ngo — 2018-11-27T10:50:01.130Z

Last month I attended a talk by David Runciman, the author of a recent book called How Democracy Ends. I was prepared for outrage-stirring and pearl-clutching, but was pleasantly surprised by the quality of his arguments, which I’ve summarised below, along with my own thoughts on these issues. Note, however, that I haven’t read the book itself, and so can’t be confident that I’ve portrayed his ideas faithfully.

Which lessons from history?

Many people have compared recent populist movements with the stirrings of fascism a century ago. And indeed it’s true that a lot of similar rhetoric being thrown around. But Runciman argues that this is one of the least interesting comparisons to be made between these two times. Some things that would be much more surprising to a denizen of the early 20th century:

Significant advances in technology
Massive transformations in societal demographics
Very few changes in our institutions

The last of those is particularly surprising in light of the first two. Parliamentary democracies in the Anglophone world have been governed by the same institutions - and in some cases, even the same parties - for centuries. Continental European democracies were more disrupted by World War 2, but have been very stable since then, despite the world changing in many ways overall. That’s true even for institutions that are probably harmful - consider the persistence of the electoral college in the US, the House of Lords in the UK, the monarchy in Australia (despite their ardent republicanism movement), first-past-the-post voting systems in many countries, and so on. (In fact, Runciman speculates that Americans voted for Trump partly because of how much confidence they had in the durability of their institutions - a confidence which so far seems to have been well-founded.)

So history gives us pretty good evidence for the robustness of democratic institutions to the normal flow of time - but not to exceptional circumstances. In fact, an inability to make necessary changes may well render them more fragile in the face of sharp opposition. If and when pressure mounts, are they going to snap like the democratic institutions of 1930s Europe did? Runciman argues that they won’t, because of the nature of the demographic changes the West has seen. There are three particularly important axes of variation:

Wealth: the average person is many times wealthier than they

...
MIRI’s 2018 Fundraiser
by Malo — 2018-11-27T05:30:11.931Z

(Crossposted from the MIRI blog)

Update Nov. 28: We’ve updated this post to announce that Haskell developer Edward Kmett has joined the MIRI team! Additionally, we’ve added links to some of our recent Agent Foundations research.

Our donors showed up in a big way for Giving Tuesday! Through Dec. 29, we’re also included in a matching opportunity by professional poker players Dan Smith, Aaron Merchak, Matt Ashton, and Stephen Chidwick, in partnership with Raising for Effective Giving. As part of their Double-Up Drive, they’ll be matching up to $200,000 to MIRI and REG. Donations can be made either on doubleupdrive.com, or by donating directly on MIRI’s website and sending your donation receipt to receipts@doubleupdrive.com. We recommend the latter, particularly for US tax residents (MIRI is a 501(c)(3) organization).

MIRI is running its 2018 fundraiser through December 31! Our progress:

Donate now

MIRI is a math/CS research nonprofit with a mission of maximizing the potential humanitarian benefit of smarter-than-human artificial intelligence. You can learn more about the kind of work we do in “

Ensuring Smarter-Than-Human Intelligence Has A Positive Outcome

” and “Embedded Agency.”

Our funding targets this year are based on a goal of raising enough in 2018 to match our “business-as-usual” budget next year. We view “make enough each year to pay for the next year” as a good heuristic for MIRI, given that we’re a quickly growing nonprofit with a healthy level of reserves and a budget dominated by researcher salaries.

We focus on business-as-usual spending in order to factor out the (likely very large) cost of moving to new spaces in the next couple of years as we continue to grow, which introduces a high amount of variance to the model.[1]

My current model for our (business-as-usual, outlier-free) 2019 spending ranges from $4.4M to $5.5M, with a point estimate of $4.8M—up from $3.5M this year and $2.1M in 2017. The model takes as input estimated ranges for all our major spending categories, but the overwhelming majority of the variance comes from the number of staff we’ll add to the team.[2]

In the mainline scenario, our 2019 budget breakdown looks roughly like this:

In this scenario, we currently have ~1.5 years of reserves on hand. Since we’ve raised ~$4.3M...
Is Science Slowing Down?
by Scott Alexander — 2018-11-27T03:30:01.516Z

[This post was up a few weeks ago before getting taken down for complicated reasons. They have been sorted out and I’m trying again.]

Is scientific progress slowing down? I recently got a chance to attend a conference on this topic, centered around a paper by Bloom, Jones, Reenen & Webb (2018).

BJRW identify areas where technological progress is easy to measure – for example, the number of transistors on a chip. They measure the rate of progress over the past century or so, and the number of researchers in the field over the same period. For example, here’s the transistor data:

This is the standard presentation of Moore’s Law – the number of transistors you can fit on a chip doubles about every two years (eg grows by 35% per year). This is usually presented as an amazing example of modern science getting things right, and no wonder – it means you can go from a few thousand transistors per chip in 1971 to many million today, with the corresponding increase in computing power.

But BJRW have a pessimistic take. There are eighteen times more people involved in transistor-related research today than in 1971. So if in 1971 it took 1000 scientists to increase transistor density 35% per year, today it takes 18,000 scientists to do the same task. So apparently the average transistor scientist is eighteen times less productive today than fifty years ago. That should be surprising and scary.

But isn’t it unfair to compare percent increase in transistors with absolute increase in transistor scientists? That is, a graph comparing absolute number of transistors per chip vs. absolute number of transistor scientists would show two similar exponential trends. Or a graph comparing percent change in transistors per year vs. percent change in number of transistor scientists per year would show two similar linear trends. Either way, there would be no problem and productivity would appear constant since 1971. Isn’t that a better way to do things?

A lot of people asked paper author Michael Webb this at the conference, and his answer was no. He thinks that intuitively, each “discovery” should decrease transistor size by a certain amount. For example, if you discover a new material that allows transistors to be 5% smaller along one dimension, then you can fit 5% more transistors on your chip whether there were a hundred there before or a million. Since the relevant factor is discoveri...
Bodega Bay: workshop
by KatjaGrace — 2018-11-27T03:20:01.290Z

What was the workshop like? Well there was really amazing carpet. I don’t know how relevant this is, but a lot of people commented on it.

Another thing that was salient to me—and many of the participants—was these things called ‘back jacks’. They are like adjustable chairs, without arms or legs.

Some part of me believes that with the right carpet and back jack, my life could be at least twenty percent better. Here is a picture from Amazon, which is on good terms with this part of me:

There were also lots of sessions that I went to, about things like how to be curious, or trade tastes with other people. Then they eventually started talking about AI, after which I started avoiding the classes. There is something nice about avoiding classes in a beautiful place, among people actively and idealistically doing things.

Before the AI bits, I went to a class called something like ‘seeking IPC’, which they said was not well named, and that if I could name it better, that would be helpful. So I payed attention to what it was about, to this end. As far as I can tell it was about how you should be curious all the time. For instance, if instead of being annoyed, one is curious, then one might get to find out the fascinating answer to the question of what the hell is going on inside the other person’s head.

(I spent the next day feeling annoyed with everything. I considered being curious about this, but found that I definitely did not want to. I did have a lot of second level curiosity about why I was so incurious.)

I found the continuing line of thought on curiosity to be fascinating and radically perspective-changing, but struggle to explain my thoughts in a way where they don’t seem obvious. This might be because they are obvious. I claim that this doesn’t undermine the importance or difficulty of realizing them. The problem with obvious things often is that you don’t realize that you have to realize them, because you assume that you already know them because they are obvious. Anyway, I’m going to keep the actual point for another time when I figure out how to explain really obvious things to people. But indescribable revelation is right up there with nice carpet and back jacks for making a good workshop I think.

I went to some classes on double crux. Double crux is basically the idea that when you are having an argument with someone, you should both try to figure out what...
Alignment Newsletter #34
by Rohin Shah — 2018-11-26T23:10:03.388Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through the database of all summaries.

Highlights

Scalable agent alignment via reward modeling (Jan Leike): This blog post and the associated paper outline a research direction that DeepMind's AGI safety team is pursuing. The key idea is to learn behavior by learning a reward and a policy simultaneously, from human evaluations of outcomes, which can scale to superhuman performance in tasks where evaluation is easier than demonstration. However, in many cases it is hard for humans to evaluate outcomes: in this case, we can train simpler agents using reward modeling that can assist the human in evaluating outcomes for the harder task, a technique the authors call recursive reward modeling. For example, if you want to train an agent to write a fantasy novel, it would be quite expensive to have a human evaluate outcomes, i.e. rate how good the produced fantasy novels are. We could instead use reward modeling to train agents that can produce plot summaries, assess prose quality and character development, etc. which allows a human to assess the fantasy novels. There are several research challenges, such as what kind of feedback to get, making it sufficiently sample efficient, preventing reward hacking and unacceptable outcomes, and closing the reward-result gap. They outline several promising approaches to solving these problems.

Rohin's opinion: The proposal sounds to me like a specific flavor of narrow value learning, where you learn reward functions to accomplish particular tasks, rather than trying to figure out the "true human utility function". The recursive aspect is similar to iterated amplification and debate. Iterated amplification and debate can be thought of as operating on a tree of arguments, where each node is the result of considering many child nodes (the considerations that go into the argument). Importantly, the child nodes are themselves arguments that can be decomposed into smaller considerations. Iterated amplification works by learning how to compose and decompose nodes from children, while debate works by having humans evaluate a particular path in the argument tree. Recursive reward modeling instead uses reward modeling to train agents that can help evaluate outcomes on the task of interest. This seems less recursive to me, since the suba...