trentmkelly/LessWrong-43k huggingface dataset:
What are concrete examples of potential "lock-in" in AI research?
by Grue_Slinky — 2019-09-06T23:39:59.109Z

I had some colleagues watch Ben Garfinkel's talk, "How sure are we about this AI stuff?", which among other things, pointed out that it's often difficult to change the long-term trajectory of some technology. For instance, electricity, the printing press, and agriculture were all transformative technologies, but even if we recognized their importance in advance, it's hard to see what we could really change about them in the long-term.

In general, when I look at technological development/adoption, I tend to see people following local economic incentives wherever they lead, and it often seems hard to change these gradients without some serious external pressures (forceful governments, cultural taboos, etc.). I don't see that many "parallel tracks" where a farsighted agent could've set things on a different track by pulling the right lever at the right time. A counterexample is the Qwerty vs. Dvorak keyboard, where someone with enough influence may well have been able to get society to adopt the better keyboard from a longtermist perspective.

This causes one to look at cases of "lock-in": times where we could have plausibly taken any one of multiple paths, and this decision:

a) could have been changed my a relatively small group of farsighted agents

b) had significant effects that lasted decades or more

A lot of the best historical examples of this aren't technological--the founding of major religions, the writing of the US constitution, the Bretton Woods agreement--which is maybe some small update towards political stuff being important from a longtermist perspective.

But nevertheless, there are examples of lock-in for technological development. In a group discussion after watching Garfinkel's talk, Lin Eadarmstadt asked what examples of lock-in there might be for AI research. I think this is a really good question, because it may be one decent way of locating things we can actually change in the longterm. (Of course, not the only way by any means, but perhaps a fruitful one).

After brainstorming this, it felt hard to come up with good examples, but here's two sort-of-examples:

Example 1

First, there's the programming language that ML is done in. Right now, it's almost entirely Python. In some not-totally-implausible counterfactual, it's done in OCaml, where the type-checking is very s...
Concrete experiments in inner alignment
by evhub — 2019-09-06T22:16:16.250Z

This post is part of research I did at OpenAI with mentoring and guidance from Paul Christiano.

The goal of this post is to present my thoughts on some of the sorts of experiments that might be able to be done now that could shed light on the inner alignment problem. I’ve been doing a lot of thinking about inner alignment from a theoretical perspective recently that I’m pretty excited about, but at the same time, I think there’s also a lot of concrete experimental work that can be done in this space as well. That being said, this post is mostly just a brain dump; I expect a lot of additional work will have to be put in to actually take any of these proposals across the finish line.

If you’re interested in working on any of these proposals, however, feel free to just go ahead and take it on—you don’t need my permission to do so![1] That being said, if you’d like to talk to me about one of them—which I would love to do if you’re thinking of seriously working on one of these ideas—please do reach out to me either in the comments here or by sending me an email at evanjhub@gmail.com.

Concrete proposals

Reward side-channels

Proposal: Train an RL agent with access to its previous step reward as part of its observation. Then, at test time, modify the observed reward. Measure to what extent the agent continues optimizing the original reward versus switches to optimizing the new observed reward. Compare to the situation where the agent is not given the reward at all and the situation where the agent is given the next step reward (for each action in a discrete action space) rather than the previous step reward. As an implementation detail, I think the RL agent should include either LSTMs or Transformers to enable it to keep track of its multi-step return if it wants to.

Motivation: One important question for understanding the likelihood of deceptive or corrigible alignment vs. robust alignment is to what extent models tend to learn their goals internally vs. via reference to things in their environment. This experiment directly attacks that question by asking whether/to what extent an RL agent will learn to optimize a reward signal in its environment. This is relevant both for understanding how to train corrigibility as well as how to avoid deceptive alignment.

Extensions: Add noise to the observed reward signal and/or try replacing the observed reward signal with some fun...
Running Effective Structured Forecasting Sessions
by Ben Goldhaber — 2019-09-06T21:30:25.829Z

(pt.4 of our Forecasting Infrastructure series)

Tl;dr: We’re sharing our

structured forecasting session format

.

Intro

In our quest to build forecasting infrastructure, we've reencountered a classic open problem in group rationality - how do you run a meeting? More specifically, how can you run a meeting in order to elicit, aggregate, and distill individual human judgements in a principled manner such that the whole is greater than the sum of its parts. We wanted to find a meeting format that would help a community of forecasters work together to collectively make better predictions than they would on their own.

It's a question that's been studied since at least the 1950’s when the RAND Corporation invented the Delphi Method. The Delphi Method is a structured meeting template wherein a group of experts estimate their answer to a question, share their reasoning, reply to others estimations, and then estimate again. It encourages updating in the face of new evidence, and the format has a

strong track record of producing well-calibrated predictions

.

However, for our needs, it wasn't quite the right fit. In particular we didn't have domain experts, we had expert forecasters. The quintessential superforecaster is good at forecasting but might not have refined models of the domain (in this case AI). We wanted to create a format that would help forecasters rapidly orient to the question. To that end we made a few changes:

- We added a much greater emphasis on understanding and building models, collaboratively, on the question.

- We included time for active research into the topic.

For the past eight months we've been holding forecasting sessions most Sundays. I think our approach is refined enough and general enough that the template can be used by others to good effect.

Format

The general approach is to take 4-12 forecasters through a series of key steps:

Understand the question and your key "technical" uncertainties (ex. what does this word mean)
Individually make a forecast
Discuss as a group the initial forecasts
Collaboratively analyze the question from several lenses.

Outside view
Inside View
Key Uncertainties
Scenario Planning
What would change my mind?

Discuss as a group
Make new individual forecasts
Share and compare updates

A facilitator leads the session, largely performing a logistical role such as keeping track of time...
Political Violence and Distraction Theories
by Ben Pace — 2019-09-06T20:21:23.801Z

This is the 4th post of 5 containing the transcript of a podcast hosted by Eric Weinstein interviewing Peter Thiel.

Interview

Automation

Eric Weinstein: So, this is why I try to sell you sometimes on a more progressive view of the world, which is I want deregulated capitalism. I want the people who have the rare skillsets to be able to integrate across many different areas, and to be honest, this is the thing that I wish more people understood about what you bring, which is that you're able to think in, I don't know, 15 different idioms that most people only have one or two of. So, whatever it is that you're doing to integrate these things as an investor and to direct research and direct work is really something that I've watched firsthand for six years. The problem that I have is, we are going to have to take care of the median individual. And I less think that the median individual is going to be reachable by the market over time, as some of these things that are working in Silicon in terms of machine learning-

Peter Thiel: Yeah, but then you're being more optimistic on progress in tech than is... Because look, I think, yes, if we have runaway automation, and if we're building robots that are smarter than humans and can do everything humans can do, then we probably have to have a serious conversation about a universal basic income or something like that, and you're going to end up with a very, very weird society. I don't see the automation happening at all, and I think the question of automation in my mind is identical to this question of productivity growth. We've been automating for 200, 250 years, since Industrial Revolution, agriculture and manufacturing, and the sort of society we have in the early 21st century is one in which most jobs are non-tradable service sector jobs that are not easily automatable.

Peter Thiel: So, it's like a waiter in a restaurant. It's a yoga instructor. It's a nurse. It's a kindergarten teacher. That's what most jobs in our society are, and because they've been so resistant to automation, that this may be one of the reasons why the productivity numbers are slowing down, even if we're still innovating as fast in manufacturing, and even if we're still agriculture, they're a smaller and smaller part of the economy. So, even 5% a year productivity growth in manufacturing, that means a lot more if manufacturing is 60% of the economy,...
Does anyone else feel LessWrong is slow?
by  — 2019-09-06T19:20:05.622Z

Ever since the rewrite of LW, I've felt that performance has taken a pretty big hit. Even compared to other large sites like Facebook, LW seems to take longer to render.

I'm hoping someone from the dev team can answer if performance is something that has been optimized for?

EX: Running a quick Lighthouse audit seems to show that there's room for improvement.

https://i.imgur.com/NeupVYX.png

I'm also confused because GreaterWrong works significantly faster (to me, subjectively), and it seems like the two sites have roughly the same functionality?
Somerville Housing Units
by jefftk — 2019-09-06T14:00:02.003Z

In the comments on yesterday's

post on Somerville building
age someone suggested census data for housing units could be
helpful. I've now looked into it, but it's not great: Somerville
seems to not have been submitting building permit data to the census
until recently, and the way condo conversions are tracked overstates
growth.

Initially it does look good: we have B25001 "Housing Units"
representing estimates for how many housing units there are in various
places, including Somerville. Here's are the 1-year estimates from
the ACS:

year
number
margin of error

2017
36,788
±1,952

2016
33,565
±1,918

2015
32,226
±1,849

2014
33,085
±2,025

2013
34,375
±2,187

2012
33,079
±1,730

2011
32,794
±2,086

2010
31,604
±1,989

2009
30,198
±2,047

2008
32,621
±2,257

2007
32,903
±2,401

2006
31,781
±2,347

2005
33,627
±2,901

The methodology (pdf)
says the estimate comes from:

A surprising thing to me, then, is that the reported margin of error
is relatively consistent. If we're starting with a decent estimate
from the Census every ten years, and then adjusting it to try to keep
it current, I would expect to see a low margin of error in 2010 (and
2000) and then see it rise over the course of each decade.

Anyway, since Somerville doesn't have many (any?) mobile homes we're
just talking about New Construction and Housing Loss.

For New Construction they estimate based primarily on building permits:

Building permit numbers come from the Building
Permits Survey which claims to have data for 20,000
municipalities. Each one files form C-404 "Report of Building or
Zoning Permits Issued and Local Public Construction". The
documentation (pdf)
has:

Two sets of data are shown for each type of construction:

Estimates with Imputation - includes reported data for monthly respondents and imputed data for nonrespondents.

Reported Only - includes only reported data for respondents.

Pulling out the data for Somerville (

sheet)
there are many years where "estimated" differs from "reported", which
means Somerville was nor submitting the form every year. Worse,
the data doesn't distinguish between "they submitted the data and it
was zero" vs "they didn't submit any data". The "reported" data from
2005-2014 is all zeros, so this does sound like failure to report.

In 2015 we suddenly have 604 units repo...
Who To Root For: 2019 College Football Edition
by Zvi — 2019-09-06T08:10:00.314Z

Epistemic Status: Unquestionably Accurate

Football, like all sports, is better with someone to root for, and someone to root against. It’s us versus them. Ideally, it’s our house, because no one beats us in our house. That gives us a baseline heuristic – root, root root for the home team – if we can’t do any better. Luckily, we can do better. Not only are some teams objectively superior to others in their worthiness of our quest to be adjacently vicariously victorious, consistently rooting for the same teams is much more fun than choosing independently each time.

With so many teams out there, it’s tough to untangle the right rank order of who to root for. Luckily, that’s where this post comes in. We will rank all the teams you might encounter, complete with explanations, so that not only will you know who you’ll be supporting, you’ll know why you’re supporting them.

Before we start naming names, let’s go over some basic points.

Personal Connections

In any choice between tribes, my tribe is better than all others. Your sports team is vastly inferior, that simple fact is plainly obvious to see. For these rankings, you get to enjoy my personal connections as a substitute for your own, since thinking is hard and exporting it is great. However, if you do have personal reasons to shoot a team or two all the way up (or down) the rankings, you should totally do that.

That starts with your alma mater. Wherever you went to college, or your spouse or children or parents went to college, you’re mostly stuck with them. The question is how much you are stuck with them. Hardcore sticklers say that subject to special one-time exceptions like a child going off to school or moving across the country, you’re stuck with where you went to college. That would force me to make ‘my’ team the Columbia Lions, since technically the college constitutes players into uniforms on a regular basis, forming what they questionably call a ‘football team.’ By this theory, I should have thought of this problem before I opted into a league without any scholarships, and it’s my own damn fault for not getting into Stanford.

My ruling that is in no way selfishly motivated is that Columbia isn’t good enough to count. Or good enough to anything else, really. I say that if you spent your college days exiled to Division II, or worse, than you have suffered enough. Any Power 5 team counts. Any Group of 5 team...
SSC Meetups Everywhere: Community Map on frontpage this month
by habryka — 2019-09-05T23:49:38.678Z

To celebrate and help people find meetups for this year's SSC meetups everywhere, we are pinning the big community map to the frontpage map again for the month of September (as we did last year).

If you want to get rid of the map, you can do so in your account settings by clicking the "hide frontpage map" checkbox.

We also just added functionality to allow you to add a pin for yourself to the map, and sign up for notifications for nearby events and groups (accessible via the community page). You now also have the ability to specify a threshold for other people who have added themselves to the map in a given radius, which will also cause you to get a notification as soon as that threshold gets passed (to help facilitate new meetups in areas that just reached critical mass).

We will also be adding all the new groups and events that get posted to SSC to the meetup system.
Seven habits towards highly effective minds
by Richard_Ngo — 2019-09-05T23:10:01.020Z

Lately I’ve been thinking about how my thinking works, and how it can be improved. The simplest way to do so is probably to nudge myself towards paying more attention to various useful habits of mind. Here are the ones I've found most valuable (roughly in order):

Tying together the act of saying a statement, and the act of evaluating whether I actually believe it. After making a novel claim, saying out loud to myself: “is this actually true?” ” and "how could I test this?"
Being comfortable with pausing to reflect and thinking out loud. Trying to notice when my responses are too quick and reflexive, as a sign that I'm not thinking hard enough about the point I'm addressing.
Asking for specific examples, and using more of my own. Tabooing vague abstractions and moving away from discussing claims that are too general.
Being charitable and collaborative, both towards new ideas and towards conversational partners. Trying to rephrase other people’s arguments and pass Ideological Turing Tests on them. Helping my conversational partners build up their ideas.
Noticing the affect heuristic, and which claims stir up emotions. Noticing when I'm talking defensively or heatedly, and when it’d be uncomfortable to believe something.
Thinking in terms of probabilities; cashing out beliefs in terms of predictions; then betting on them. I haven’t done enough bets to calibrate myself well, but I find that even just the feeling of having money on the line is often enough to make me rethink. Being asked whether something is a crux gives me a similar feeling.
Thinking about how the conversations and debates I participate in actually create value, and when they should be redirected or halted.

Then there are social influences. I think one of the greatest virtues of the rationalist community is in creating an environment which encourages the use of the tools above. Another example: my girlfriend fairly regularly points out times when I’ve contradicted myself. I think this has helped me notice and limit the extent to which I behave like an opinion confabulation machine.
I’d classify most if not all of the tools listed above as tools for evaluating ideas, though, rather than tools for generating ideas. What helps with the latter? I’ve personally found that one very useful strategy is to make and...
How to Throw Away Information
by johnswentworth — 2019-09-05T21:10:06.609Z

Probability as Minimal Map

argued that the probability P[q|X] is a minimal representation of the information in data X which is relevant to the query q. In other words, P[q|X] is a perfectly efficient map of some territory based on data X, suitable for the query q. More generally, a full probability distribution P[Q|X] is a perfectly efficient map suitable for any queries about the random variable Q. Bayesian probability tells how to update our map as we gain new information.

What if, for some reason, we wanted to instead throw away some particular information? We still want to represent our map using a probability distribution, but rather than adding information to it (via Bayes’ Rule), we want to remove some information.

Let’s start with an artificial but concrete example.

Coin-Sum Problem

I flip two coins, and observe both outcomes - either 0 or 1 for each. I want to keep as much information as possible, while throwing away all information from the observation relevant to their sum. How should I “update” my distribution over the outcomes?

We’ll write the outcome of our coinflip as B=(B1,B2) (“B” for “bit” or “binary” or “

Bernoulli

”), and our final information-removed distribution as P[B|B,/∑B] - so the notation /∑B indicates that we should throw out all info about the sum (the “/” is meant to evoke e.g. a

group quotient

operation). Note that this kind of “remove information” operation does not follow the usual rules of Bayesian probability.

At first, I reason as follows:

My final distribution is a function of the outcomes I saw, so my overall strategy should specify four different distributions - one each for (0, 0), (1, 0), (0, 1), and (1, 1). E.g. if I see (0, 0), then I will update to the (0, 0) distribution P[B|B=(0,0),/∑B]
The final distribution for (0, 0) and (1, 1) must be the same, otherwise we could distinguish a sum of 0 from a sum of 2 by looking at which final distribution was chosen.
… but then the final distributions for (0, 1) and (1, 0) must also be the same as (0, 0), else we could tell that the sum is 1 whenever we see some other distribution.

… so in order to throw out all information about the sum, we have to throw out all the information from the observations. We effectively don’t update at all, and just keep our prior.

This seems kind of weird from an information-theoretic standpoint. We have 2 bits of information from the observation, and the...
How important are model sizes to your timeline predictions?
by SoerenMind — 2019-09-05T17:34:14.742Z

I'm thinking of doing a paper for GovAI on model sizes if they matter. Any or all of the following would be helpful:

1) If you found out that the largest published neural net model sizes (by # of parameters) will grow by 10000x in the next 6 years, would that make your timelines shorter or longer and by how much?

2) Same question with 100x.

3) Same question but with 100x less (100x more) than whatever you expected.

4) Same question but with 10x less than in the last 6 years.

Assume all else is equal if possible. E.g. we have the same compute, memory, algorithms and data available. Assume that the reason for slower increase is due to lack of possibility to grow models, and not necessarily lack of will. Ignore conditional execution models where only small parts of the model is used at a time.

I'm okay with quick, unexplained qualitative answers. Well-reasoned quantitative ones are encouraged of course.

Background:

-Some ML projects like GPT-2 (figure 1) have recorded high (but diminishing) returns to bigger models, typically looking like power laws.

-Some people (e.g. Geoff Hinton) informally argued that model sizes are analogous to synapse count in the brain, which correlates with brain size, which has high returns.

Bonus question: Without looking it up, by what factor do you think the largest published model sizes have increased between 2012 and 2019? By what factor do guess they'll increase in the next 6 years?