trentmkelly/LessWrong-43k huggingface dataset:
Covid 7/30: Whack a Mole
by Zvi — 2020-07-30T15:50:04.756Z

Last week: Covid 7/23: The Second Summit

It seems clear that positive test rates, and with them new infections, have peaked in the Southern epicenters.

Unless and until something changes, forward looking risk will continue to improve in the South and West. The Midwest and Northeast, however, are still getting worse.

The two additional big unknowns of school and winter remain on the horizon. So does the election. So it’s very much too early for celebrations.

This week’s worst Covid news was the outbreak among the Florida Marlins, and its implications going forward. Oh, and also the number of deaths went substantially up, but that was baked in weeks ago. The rate of increase was a little faster than I expected, but that was only because I adjusted based on last week’s rate of increase being slower than expected, and some of it is (good and appropriate) changes in reporting methodology.

The other news this week was several efforts to point out the obvious truth of heterogeneity, that different people are different and do different things in different places and react differently to exposure, and that this might matter. I wrote about one such effort here and there are more that have now been pointed out. I hope to get to the others soon.

In non-Covid news, GPT-3 continues to be the scariest, most exciting and most fun thing to come around in a long time. Alas, I continue to not have direct access, so I’m forced to play with AI Dungeon. Which is already super awesome, but also will eventually inject vampires and demons and rituals into every scenario even if you literally enter “vampires do not exist” as the sole content of the /remember command. When I get a chance I’ll try to write up what I’ve found out so far.

SlateStarCodex continues to be back, but Scott’s new output is limited to really horrible puns on Twitter. Until the blog is back in production, the saga is not yet over.

Let’s run the numbers.

Positive Test Counts

Date
WEST
MIDWEST
SOUTH
NORTHEAST

May 21-May 27
23979
39418
42977
37029

May 28-June 3
32200
31504
50039
33370

June 4-June 10
35487
24674
55731
22693

June 11-June 17
41976
22510
75787
17891

June 18-June 24
66292
26792
107221
15446

June 25-July 1
85761
34974
163472
16303

July 2-July 8
103879
40139
202863
18226

July 9-July 15
108395
53229
250072
20276

July 16-July 22
117506
57797
265221
20917

July 23-July 29
110219
67903
24

...
Is the work on AI alignment relevant to GPT?
by Richard_Kennaway — 2020-07-30T12:23:56.842Z

I see a lot of posts go by here on AI alignment, agent foundations, and so on, and I've seen various papers from MIRI or on arXiv. I don't follow the subject in any depth, but I am noticing a striking disconnect between the concepts appearing in those discussions and recent advances in AI, especially GPT-3.

People talk a lot about an AI's goals, its utility function, its capability to be deceptive, its ability to simulate you so it can get out of a box, ways of motivating it to be benign, Tool AI, Oracle AI, and so on. Some of that is just speculative talk, but there does appear to be real mathematics going on, for example on embedded agency. But when I look at GPT-3, even though this is already an AI that Eliezer finds alarming, I see none of these things. GPT-3 is a huge model, trained on huge data, for predicting text. That is not to say that it cannot be understood in cognitive terms, but I see no reason to expect it to be. It is at least something that would have to be demonstrated before any of the formalised work on AI safety would be relevant.

People speculate that bigger and better versions of GPT-like systems may give us some level of real AGI. Can systems of this sort be interpreted as having goals, intentions, or any of the other cognitive and logical concepts that the AI discussions are predicated on?
Would a halfway copied brain emulation be at risk of having different values/identity?
by Ghatanathoah — 2020-07-30T05:43:30.772Z

When I was thinking about the concept of human brain emulation recently, a disturbing idea occurred to me. I have never seen anyone address it, so I suspect it is probably caused by my being deeply confused about either human neurobiology or computer science. I thought I'd ask about it in the hopes someone more informed would be able to explain it and the idea could stop bothering me:

Imagine that a brain emulation is in the process of being encoded into a storage medium. I don't think that it is relevant whether the copy is being made from an organic brain or an existing emulation. Presumably it takes some amount of time to finish copying all the information onto the storage media. If the information is about a person's values or personality, and it is only halfway copied, does that mean for a brief moment, before the copying process is complete, that the partial copy has very different personality or values from the original? Are the partially copied personality/values a different, simpler set of personality/values?

Presumably the copy is not conscious during the copying process, but I don't think that affects the question. When people are unconscious they still have a personality and values stored in their brain somewhere, they are just not active at the moment.

I find this idea disturbing because it implies that emulating any brain (and possibly copying de novo AI as well) would inevitably result in creating and destroying multiple different personality/value sets that might count as separate people in some way. No one has ever brought this up as an ethical issue about uploads as far as I know (although I have never read "Age of Em" by Robin Hanson), and my background is not tech or neuroscience, so there is probably something I am missing .

Some of my theories of things I am missing include:

I am naively viewing personality/values as being stored in the brain in a simple, list-like format. For example, I may be imagining that someone who likes reading and jogging is being copied, and there is a brief period where the copy is someone who only likes reading because "jogging" has not been copied yet. In reality personality/values are probably distributed in the brain in some complex way that would not work like that. If they are only partially copied the emulation would simply crash, or not have values (be unmot

...
Attention is your scarcest resource
by benkuhn — 2020-07-30T01:00:13.705Z

Like many people, I have most of my best ideas in the shower.

This is sometimes annoying: I could use more than one shower’s worth of good ideas a day, but I’d rather not end up as a shrivelled yet insightful prune. Mostly, though, shower ideas are the incentive that keeps me smelling okay, so I grudgingly accept the constraint.

The time when it was most constraining was the first time I became a manager. I only had a few reports, so managing them wasn’t a full-time job. But I was very bad at it, and so it should have been what I spent all my shower insights on.

Unfortunately, I was spending my non-management time on programming. And even if I tried to use my showers to think about my thorny and awkward people issues, my mind somehow always wandered off to tackle those nice, juicy software design problems instead.

Management Attempt #1 didn’t go very well; being distracted by programming, I made lots of embarrassing mistakes that made my reports’ lives harder, which in turn made me feel stressed and guilty. I ended up bouncing back to being an individual contributor after a few months.

That experience of mine resonates strongly with Byrne Hobart’s observation about focus in knowledge work:

The output of knowledge workers is extremely skewed based on focus. The productivity tiers seem to be:

<10% focused on the job at hand: meaningful risk of getting fired.

10-50% focus: “meets expectations,” gets regular raises.

50%+ focus: superstar, 10x engineer, destined for greatness.

“50%+ focus” is roughly when something becomes the top idea in your mind. It’s when you start caring enough to think about it in the shower. It’s when you start habitually asking “how could this go faster?” It’s when you get relentlessly resourceful. It’s around when you start annoying your coworkers and/or significant other, although that part is avoidable with practice.

Most importantly, you can only be 50%+-focused on one thing at a time—or zero, in bad cases. That makes it critical to conserve your attention, so that you can spend it on what matters.

I’ve found a few different strategies that help me with this:

Care viscerally

It’s hard to recruit 50%+ of your brain to work on something just because you feel like you “should” do it. Your superego might listen to this newfangled “logic” and “reason” stuff, but the monkey-era parts of your brain can’t be fooled so easily. To be reli...
Learning the prior and generalization
by evhub — 2020-07-29T22:49:42.696Z

This post is a response to Paul Christiano's post “Learning the prior.”

The generalization problem

Generally, when we train models, we often end up deploying them in situations that are distinctly different from those they were trained under. Take, for example, GPT-3. GPT-3 was trained to predict web text, not serve as a dungeon master—and the sort of queries that people present to AI dungeon are quite different than random web text—but nevertheless GPT-3 can perform quite well here because it has learned a policy which is general enough that it continues to function quite effectively in this new domain.

Relying on this sort of generalization, however, is potentially quite troublesome. If you're in a situation where your training and deployment data are in fact independently and identically distributed (i.i.d.), you can produce all sorts of nice guarantees about the performance of your model. For example, in an i.i.d. setting, you know that in the limit of training you'll get the desired behavior. Furthermore, even before the limit of training, you know that validation and deployment performance will precisely track each other such that you can bound the probability of catastrophic behavior by the incidence of catastrophic behavior on the validation data.

In a generalization setting, on the other hand, you have no such guarantees—even in the limit of training, precisely what your model does off-distribution is determined by your training process's inductive biases. In theory, any off-distribution behavior is compatible with zero training error—the only reason machine learning produces good off-distribution behavior is because it finds something like the simplest model that fits the data. As a result, however, a model's off-distribution behavior will be highly dependent on exactly what the training process's interpretation of “simpler” is—that is, its inductive biases. And relying on such inductive biases for your generalization behavior can potentially have catastrophic consequences.

Nuances with generalization

That being said, the picture I've painted above of off-distribution generalization being the problem isn't quite right. For example, consider an autoregressive model (like GPT-3) that's just trained to learn a particular distribution. Then, if I have some set of training data Xtrain and a new data point x′, there's no test you can do to determine whethe...
What Failure Looks Like: Distilling the Discussion
by Ben Pace — 2020-07-29T21:49:17.255Z

The comments under a post often contains valuable insights and additions. They are also often very long and involved, and harder to cite than posts themselves. Given this, I was motivated to try to distill some comment sections on LessWrong, in part to start exploring whether we can build some norms and some features to help facilitate this kind of intellectual work more regularly. So this is my attempt to summarise the post and discussion around What Failure Looks Like by Paul Christiano.

Epistemic status: I think I did an okay job. I think I probably made the most errors in place where I try to emphasise concrete details more than the original post did. I think the summary of the discussion is much more concise than the original.

What Failure Looks Like (Summary)

On its default course, our civilization will build very useful and powerful AI systems, and use such systems to run significant parts of society (such as healthcare, legal systems, companies, the military, and more). Similar to how we are dependent on much novel technology such as money and the internet, we will be dependent on AI.

The stereotypical AI catastrophe involves a powerful and malicious AI that seems good but suddenly becomes evil and quickly takes over humanity. Such descriptions are often stylised for good story-telling, or emphasise unimportant variables.

The post below will concretely lay out two ways that building powerful AI systems may cause an existential catastrophe, if the problem of intent alignment is not solved. This is solely an attempt to describe what failure looks like, not to assign probabilities to such failure or to propose a plan to avoid these failures.

There are two failure modes that will be discussed. First, we may increasingly fail to understand how our AI systems work and subsequently what is happening in society. Secondly, we may eventually give these AI systems massive amounts of power despite not understanding their internal reasoning and decision-making algorithms. Due to the massive space of designs we'll be searching through, if we do not understand the AI, this will mean certain AIs will be more power-seeking than expected, and will take adversarial action and take control.

Failure by loss of control

There is a gap between what we want, and what objective functions we can write down. Nobody has yet created a function that when maximised perfectly describes what...
New Paper on Herd Immunity Thresholds
by Zvi — 2020-07-29T20:50:01.242Z

Previously: On R0

This new paper suggests that herd immunity could be achieved with only about 10% infected rather than the typically suggested 60%-70%.

They claim this is due to differences in connectivity and thus exposure, and in susceptibility to infection. They claim that the best model fit for four European epidemics at 16%-26% for England, 9.4%-11% for Belgium, 7.1%-9.9% for Portugal, and 7.5%-21% for Spain.

This being accurate would be excellent news.

The 60%-70% threshold commonly thrown around is, of course, utter nonsense. I’ve been over this several times, but will summarize.

The 60%-70% result is based on a fully naive SIR (susceptible, infected, recovered) model in which all of the following are assumed to be true:

People are identical, and have identical susceptibility to the virus.

People are identical, and have identical ability to spread the virus.

People are identical, and have identical exposure to the virus.

People are identical, and have contacts completely at random.

The only intervention considered is immunity. No help from behavior adjustments.

All five of these mistakes are large, and all point in the same direction. Immunity matters much more than the ‘naive SIR’ model thinks. Whatever the threshold for immunity might be for any given initial reproduction rate, it’s nowhere near what the naive SIR outputs.

Often they even take the number of cases with positive tests to be the number of infections, and use that to predict forward or train their model.

This naive model is not a straw man! Such obvious nonsense models are the most common models quoted by the press, the most common models quoted by so-called ‘scientific experts’ and the most common models used to determine policy.

The effective response is some combination of these two very poor arguments:

“Until you can get a good measurement of this effect we will continue to use zero.”

“Telling people the threshold is lower will cause them to take less precautions.”

Neither of these is how knowledge or science works. It’s motivated cognition. Period.

See On R0 for more details on that.

So when I saw this paper, I was hoping it would provide a better perspective that could be convincing, and a reasonable estimate of the magnitude of the effect.

I think the magnitude they are suggesting is very reasonable. Alas, I do not think the paper is convincing.

The Model

The model involves the use of ...
What a 20-year-lead in military tech might look like
by Daniel Kokotajlo — 2020-07-29T20:10:09.303Z

I've spent way too much time speculating about near-future military technology. Here is a list of technologies I think are likely to be important to a conventional war in, say, 2040.

If you think this list shouldn't be taken seriously because I don't have official expertise, you are probably right. I think that if I became an expert I’d probably end up deleting a third of the things on this list and adding a similar number of new things. (I did run this draft past three people with relevant expertise, though!) I look forward to critiques in the comments.

If you are wondering why these speculations are relevant to LessWrong… well, it was requested, and also it’s relevant to a point I want to make later about AI. You see, the technologies on this list could probably be quickly achieved with the help of advanced-but-not-godlike-AI. The gap between these technologies and present-day military tech is an example of a “small” gap; a “merely” 20-year-by-human-standards gap. So, for example, suppose some corporation or nation gets AI systems which are good but not godlike at designing new technologies. They vastly speed up the process but don’t do anything human engineers couldn’t do in 20 years. Or, suppose there are lots of AI systems in the world but one faction has systems which are the equivalent of 20-years-by-human-standards ahead. In either case, I think, the gap would translate to an extremely large advantage, possibly enough to enable

a small group to take over the world

,

conquistador-style

.

(To be clear, while I do think AI takeover is probable, I think persuasion/politics/ideology is by far the most likely method, not military/war.)

Better command-and-control capabilities (incl. Cyber and intel)

Command & control technologies are very important. Wars are won or lost on the basis of how well each side is able to observe, predict, plan, organize, and outmaneuver the other. I think the advancements in command and control tech that are likely to happen in the next 20 years are more important than everything else on this list combined. In case this isn’t obvious, I’ll say a few things below by way of justification.

Long-ranged guided weaponry already makes it easy to blow up targets if you know where they will be when your attack arrives. Winning is all about finding the enemy before they find you. Now more than ever, being able to quickly gather loads of info...
Engaging Seriously with Short Timelines
by sapphire — 2020-07-29T19:21:31.641Z

It seems like transformative AI might be coming fairly soon. By transformative AI I just mean AI that will rapidly accelerate economic and technological progress. Of course, I am not ruling out a true singularity either. I am assuming such technology can be created using variants of current deep learning techniques.

Paul Christiano has written up arguments for a 'slow takeoff' where "There will be a complete 4-year interval in which world output doubles, before the first 1-year interval in which world output doubles.". It is unclear to me whether that is more or less likely than a rapid and surprising singularity. But it certainly seems much easier to prepare for. I don't think we have a good model of what exactly will happen but we should prepare for as many winnable scenarios as we can.

What should we do now if think big changes are coming soon? Here are some ideas:

Work on quickly useable AI safety theory: Iterated Amplification and Distillation - Assuming timelines are short we might not have time for provably safe AI. We need ai-safety theory that can be applied quickly to neural nets. Any techniques that can quickly be used to align GPT-style AI are very high value. If you have the ability, work on them now.

IDA is a good framework to bet on imo. OpenAI seems to be betting on IDA. Here is an explanation. Here is a lesswrong discussion. If you are mathematically inclined and understand the basics of deep learning now might be a great time to read the IDA papers and see if you can contribute.

Get capital while you can - Money is broadly useful and can be quickly converted into other resources in a critical moment. At the very least money can be converted into time. Be frugal, you might need your resources soon.

Besides, the value of human capital might fall. If you have a lucrative position (ex: finance or tech) now is a good time to focus on making money. The value of human capital might fall. Investing in your human capital by going back to school is a bad idea.

Invest Capital in companies that will benefit from AI technology - Tech stocks are already expensive so great deals will be hard to find. But if things get crazy you want your capital to grow rapidly. I would especially recommend hedging 'transformative aI' if you will get rich anyway if nothing crazy happens.

I am doing something like the following portfolio:

AR...
[AN #110]: Learning features from human feedback to enable reward learning
by Rohin Shah — 2020-07-29T17:20:04.369Z

Newsletter #110

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

HIGHLIGHTS

Feature Expansive Reward Learning: Rethinking Human Input (Andreea Bobu, Marius Wiggert et al) (summarized by Rohin): One goal we might have with our algorithms is that after training, when the AI system is deployed with end users, the system would be personalized to those end users. You might hope that we could use deep inverse RL algorithms like AIRL (AN #17), but unfortunately they require a lot of data, which isn’t feasible for end users. You could use earlier IRL algorithms like MCEIRL (AN #12) that require you to specify what features of the environment you care about, but in practice you’ll never successfully write down all of these features. Can we somehow get the best of both worlds?

Past work (AN #28) made progress on this front, by allowing the agent to at least detect when it is missing some feature, by checking whether the human feedback is surprisingly inefficient given the existing features. But what do you do once you detect it? The key insight of this paper is that applying a deep IRL algorithm here would be inefficient because it has to implicitly learn the unknown feature, and we can do much better by explicitly querying the human for the unknown feature.

In particular, their method Feature Expansive Reward Learning (FERL) asks the human for a few feature traces: demonstrations in which the new feature’s value monotonically decreases. For example, suppose a robot arm carrying a cup of water gets too close to a laptop, but the arm doesn’t know the feature “close to a laptop”. Then a feature trace would start with the arm close to the laptop, and move it successively further away. Given a set of feature traces, we can convert this into a dataset of noisy comparisons, where earlier states are more likely to have higher feature values than later states, and use this to train a neural net to predict the feature value (similarly to the reward model in Deep RL from Human Preferences). We can then add this to our set of features, and learn rewards over the new set of features.

They ...