trentmkelly/LessWrong-43k huggingface dataset:
Reflections on AI Timelines Forecasting Thread
by Amandango — 2020-09-01T01:42:40.349Z

It’s been exciting to see people engage with the

AI forecasting thread

that Ben, Daniel, and I set up! The thread was inspired by Alex Irpan’s AGI timeline update, and our hypothesis that visualizing and comparing AGI timelines could

generate better predictions

. Ought has been working on the probability distribution tool,

Elicit

, and it was awesome to see it in action.

14 users shared their AGI timelines. Below are a number of their forecasts overlaid, and an aggregation of their forecasts.

Comparison of 6 top-voted forecasts

Aggregation, weighted by votes

The thread generated some interesting learnings about AGI timelines and forecasting. Here I’ll discuss my thoughts on the following:

The object level discussion of AGI timelines
How much people changed their minds and why
Learnings about forecasting
Open questions and next steps

AGI timelines

Summary of beliefs

We calculated an aggregation of the 14 forecasts weighted by the number of votes each comment with a forecast received. The question wasn’t precisely specified (people forecasted based on slightly different interpretations) so I’m sharing these numbers mostly for curiosity’s sake, rather than to make a specific claim about AGI timelines.

Aggregated median date: June 20, 2047
Aggregated most likely date: November 2, 2033
Earliest median date of any forecast: June 25, 2030
Latest median date of any forecast: After 2100

Emergence of categories

I was pleasantly surprised by the emergence of categorizations of assumptions. Here are some themes in the way people structured their reasoning:

AGI from current paradigm (2023 – 2033)

GPT-N gets us to AGI
GPT-N + improvements within existing paradigm gets us to AGI

AGI from paradigm shift (2035 – 2060)

We need fundamental technical breakthroughs

Quantum computing
Other new paradigms

AGI after 2100, or never (2100 +)

We decide not to build AGI

We decide to build tool AI / CAIS instead
We move into a stable state

It’s harder than we expect

It’s hard to get the right insights
We won’t have enough compute by 2100

We can’t built AGI

There’s a catastrophe that stops us from being able to build AGI

Outside view reasoning

With 50% probability, things will last twice as long as they already have
We can extrapolate from rate of reaching past AI milestones

When sharing their forecasts, people associated these assumptions with a corresponding date interval for when we would see AGI...
A Policy for Biting Bullets
by  — 2020-08-31T21:00:33.396Z

[Cross-posted from Grand, Unified, Empty.]

I.

The CFAR Handbook has a really interesting chapter on policy-level decision-making (pages 170-173). It’s excellent, grounds much of this post, and comes with some classic Calvin & Hobbes comics; I recommend it. If you’re too lazy for that, I’ll summarize with a question: What should you do when you’ve made a plan, and then there’s a delay, and you really don’t want to do the thing you’ve planned anymore? The handbook starts with two basic perspectives:

“Look,” says the first perspective. “You’ve got to have follow-through.
You’ve got to be able to keep promises to yourself. If a little thing like a few
hours’ delay is enough to throw you off your game, there’s practically no point
in making plans at all. Sometimes, you have to let past you have the steering
wheel, even when you don’t feel like it anymore, because otherwise you’ll
never finish anything that takes sustained effort or motivation or attention.”

“Look,” says the second perspective. “There’s nothing to be gained from
locking yourself in boxes. Present you has the most information and context;
past you was just guessing at what you would want in this moment. Forcing
yourself to do stuff out of some misguided sense of consistency or guilt or
whatever is how people end up halfway through a law degree they never
actually wanted. You have to be able to update on new information and
adapt to new circumstances.”

Policy-level decision-making is the handbook’s suggested way to thread this needle:

[W]hat policy, if I followed it every time I had to make a decision like this, would strike the right balance? How do I want to trade off between follow-through and following my feelings, or between staying safe and seizing rare opportunities?

It’s obviously more work to come up with a policy than to just make the decision in the moment, but for those cases when you feel torn between the two basic perspectives, policy-level decision-making seems like a good way to resolve the tension.

II.

There is a peculiar manoeuvre in philosophy, as in life, called “biting the bullet”. Biting the bullet in life is to accept and then do something painful or unpleasant because you don’t think you have any better al...
Helping people to solve their problems
by Neel Nanda — 2020-08-31T20:41:04.796Z
Link: https://www.neelnanda.io/blog/30-debugging-others

Introduction

Humans are not automatically strategic. Our goals are complicated, the world is complicated, and actually doing things is hard. This means that the default state of the world is that you are missing out on a lot of ways of achieving your goals, both small and large. And this is normal and to be expected! We all procrastinate, neglect opportunity costs and the value of information, are over or under-confident about our own abilities, and are generally mired in a range of biases that hold us back from being effective.

One interesting side-effect of this, is that it’s often way easier to notice these mistakes and biases in other people’s heads than in mine. And it’s much easier to make progress when you have somebody else helping you! You can separate the cognitive work of actually strategising and doing, from the cognitive work of paying attention to biases and keeping yourself focused. And, as a result, I’ve noticed that I’ll moderately frequently have a conversation with a friend where I notice that they’re missing out on ways of achieving their goals, and try to help them fix this. This is a bit wordy, so I’ll henceforth refer to this as debugging their problems.

This is something I would have assumed would be annoying, but people generally seem to be grateful and appreciative! (Even after trying to account for the inevitable sampling bias) So I’ve mostly decided that this is something worth continuing doing, and a way to add value to the lives of my friends (if they want it). This post is my attempt to write up my thoughts on how to do this well, and in ways that don’t make my friends hate me!

Health warning: If you actually intend to try these ideas, I recommend explicitly asking people if they want help solving their problems! This can be a way to add a significant amount of value, but can be super annoying if they’re tired and just want to vent, or aren’t actually interested in solving the problem.

With that aside, I think this is a really valuable skill to cultivate! It can help your friends become much more effective people, and I find it often helps me to debug my own problems, by mentally framing it as giving advice to somebody else.

Giving Them Agency

The first and most important point - you are helping them to solve their problem. The goal is not to have things to...
(Humor) AI Alignment Critical Failure Table
by Kaj_Sotala — 2020-08-31T19:51:18.266Z
Link: http://sl4.org/archive/0310/7163.html

The "Friendly AI Critical Failure Table" was originally posted by Eliezer in 2003; when I ran across a mention of it somewhere, I realized that a lot of people probably haven't seen this old classic yet.

(For people not familiar with the concept, "critical failure tables" are found in some tabletop role-playing games. If you are trying to do something and fail really badly (a "critical failure"), you may be required to consult a table that contains a list of catastrophic outcomes that your screw-up might cause, rolling some dice to determine which one of them you did cause.)
Plans / prepping for possible political violence from upcoming US election?
by anon03 — 2020-08-31T18:18:55.605Z

The US presidential election is now just 2 months away! There's a (IMO) plausible argument that we might have a disputed election that leads to large-scale political violence; for details see

Vox: "Will He Go? A law professor fears a meltdown this November"
The American Interest: "The Danger of Electoral Violence in the United States"
Metaculus: If President Trump loses the 2020 election, will he concede?

I don't think it's likely, but I do think it's likely enough to make it worth some contingency planning (for US residents like me).

Like most of us, I'm not in any position to actually do anything about this problem, i.e. to reduce the probability of large-scale political violence from a disputed election. I just want to prudently plan for myself, family, friends, and/or community to get safely through whatever happens, if anything happens.

I'm curious what (if anything) people here are doing, or plan to do, or would recommend, to prepare for this contingency. Stock up on food / water / supplies? Take an extended vacation to another country, or to a lower-risk place within the country? (And where would that be?) Move assets into gold / bitcoin / international index funds / whatever? Buy weapons? Or what?

I'm especially interested to hear from people who have experience or knowledge of what large-scale political violence is actually like, from first- or second-hand experience, or from being well-informed about history and world politics (which I am not). And for people in the USA, what are your personal plans? But everyone is very welcome to answer / comment / discuss!
From GPT to AGI
by ChristianKl — 2020-08-31T13:28:35.442Z

Epistemic status: Shower thoughts / I have an idea, but not much knowledge

While the the smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads, the largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.

I would expect that with increased model size it will be possible to increase the attention field by a lot without much need for additional AI insight.

In a guide for AI dungeon there's a description of a pin item that serves as the context for the story. GPT-3 seems to be good at understanding the goals that are set in the pin item to the point that it tries to achieve them faster then the person who wrote the guide desires.

If the attention field would be larger it would allow for multiple pin items and each pin item could be even larger.

In AI dungeon GPT-3 can't influence the content of the pin items but it would be possible to give the GPT-3 the ability to use console commands to write into pin items to be able to have memory abilities similar to the short-term and middle term memory that humans have.

The ability to interact with the unix console was already shown Natural Language Shell example on the OpenAI website.

At the beginning a resulting agent could be mentored like AI dungeon has mentoring of the AI. If the AI is able to query Google via the console, I would imagine that it could be effective at many tasks.

A lot of newspaper articles could be written by such an agent who scouts the internet for other information that's published on the topic and synthesizes available information. Cyberwar could also be done by such agents.

I'd be happy to hear what other people think about such an agent.
Alarmism
by Basil Marte — 2020-08-31T12:53:42.511Z

Content warning: conjunction fallacy. Trigger warning: basilisk for anxiety.

Sources

https://everythingstudies.com/2019/03/25/the-tilted-political-compass-part-2-up-and-down/

Interestingly, I think I see the difference between leftists and liberals become more important for Americans as well — at least among the coastal, educated middle classes where outright conservatives are rare enough to make the other two turn on each other in fits of online culture warring, just like game theory would predict. In a pure Thrive environment the ways leftists and liberals agree become nothing more than background scenery and the divisions start to stand out.

https://srconstantin.wordpress.com/2018/04/24/wrongology-101/
https://slatestarcodex.com/2014/04/22/right-is-the-new-left/

Fundies – in all of their Bible-beating gun-owning cousin-marrying stereotypicalness – have so far served as the Lower Class With Which One Must Not Allow One’s Self To Be Confused. But I think that’s changing. Sorting mechanisms are starting to work so well that, at the top, the fundies just aren’t plausible.

Reasoning

Over the last decade, the coalition between socialists and liberals has visibly become increasingly tenuous in America. This suggests to me that a rearrangement, such that the two political coalitions would not be separated along the thrive-survive axis but along the cognitive (de)coupling axis, is somewhat probable within a decade (10-50%, I'm uncertain whether my crystal ball works at all). Conditional on this happening, I find it much more probable that the existing party brands would move "clockwise" (Democrat couplers, Republican decouplers) than the contrary. To my mind, the most salient trigger (or "factor that is currently missing") is a libertarian (rather than Up) Republican presidential candidate.

Where would memes develop, particularly of the hypothesized Left+Up coalition? There's already a lot of anti-capitalism, anti-decoupling (of the "math is racist" sort), and ancestry-based category-essentialism applied to people. These can be expected to be intensified once they no longer have to please the erstwhile allies, the liberals. Furthermore, the makeup of the respective coalitions would by itself create a situation where the currently (in 2020) narratively-approved ethnicities would be jo...
interpreting GPT: the logit lens
by nostalgebraist — 2020-08-31T02:47:08.426Z

This post relates an observation I've made in my work with GPT-2, which I have not seen made elsewhere.

IMO, this observation sheds a good deal of light on how the GPT-2/3/etc models (hereafter just "GPT") work internally.

There is an accompanying Colab notebook which will let you interactively explore the phenomenon I describe here.

[Edit: updated with another section on comparing to the inputs, rather than the outputs. This arguably resolves some of my confusion at the end. Thanks to algon33 and Gurkenglas for relevant suggestions here.]

[Edit 5/17/21: I've recently written a new Colab notebook which extends this post in various ways:

trying the "lens" on various models from 125M to 2.7B parameters, including GPT-Neo and CTRL
exploring the contributions of the attention and MLP sub-blocks within transformer blocks/layers
trying out a variant of the "decoder" used in this post, which dramatically helps with interpreting some models

]

overview

GPT's probabilistic predictions are a linear function of the activations in its final layer. If one applies the same function to the activations of intermediate GPT layers, the resulting distributions make intuitive sense.

This "logit lens" provides a simple (if partial) interpretability lens for GPT's internals.
Other work on interpreting transformer internals has focused mostly on what the attention is looking at. The logit lens focuses on what GPT "believes" after each step of processing, rather than how it updates that belief inside the step.

These distributions gradually converge to the final distribution over the layers of the network, often getting close to that distribution long before the end.

At some point in the middle, GPT will have formed a "pretty good guess" as to the next token, and the later layers seem to be refining these guesses in light of one another.
The general trend, as one moves from earlier to later layers, is

"nonsense / not interpretable" (sometimes, in very early layers) -->
"shallow guesses (words that are the right part of speech / register / etc)" -->
"better guesses"

...though some of those phases are sometimes absent.

On the other hand, only the inputs look like the input tokens.

In the logit lens, the early layers sometimes look like nonsense, and sometimes look like very simple guesses about the output. They almost never look like the input.
Apparently, the model does not "keep the inputs ar

...
Announcing: Progress Studies Reading Group.
by Sami_beanz — 2020-08-30T10:39:44.278Z

This is cross-posted from the EA forum. https://forum.effectivealtruism.org/posts/JBuWGZzCb9TMHNgHw/announcing-progress-studies-reading-group

Kris Gulati recently began to organize a virtual Progress Studies reading group. Everyone is welcome to attend. The group is organized on the

Progress Studies Slack

#reading-group channel, and you can find more details there.

Format

If you're unfamiliar with EA-related reading groups this post will help.

We will have a meeting once a week where one person is selected to summarise the designated reading and present that summary to the rest of the group. We then proceed to discuss that topic.

The meeting will last one hour, and most of the readings will be related to topics that are Progress related [Check this list for guidance]. The reading materials will be book chapters or journal articles. However, since this will be mostly be determined by a group vote, informal articles, podcasts, and videos are not unlikely.

Next steps

Next Saturday at 9:00 pm GMT, we will be discussing chapter one of Order Without Design: How Markets Shape Cities by Alain Bertaud, a member of the group will be presenting it, so you don't even have* to read it!

*Although, it's a really good book so maybe you should.
microCOVID.org: A tool to estimate COVID risk from common activities
by catherio — 2020-08-29T23:01:02.081Z
Link: https://microcovid.org/

This is a linkpost for a model and web tool (that I and several friends created) to quantitatively estimate the COVID risk to you from your ordinary daily activities:

This website contains three outputs of our work:

a web calculator that you can use to calculate your COVID risk (in units of microCOVIDs, a 1-in-a-million chance of getting COVID).
a white paper that explains our estimation method. EAs might be particularly interested in the footnotes throughout, and the detailed research sources section.
a spreadsheet to compute your COVID risk in more detail and to track your risk over time. EAs might find this more customizable and powerful than the web calculator.

If you have different beliefs than us and would like to use a version of the model that reflects your beliefs rather than ours, you can make modifications to your copy of the spreadsheet, or fork the repository and make a personal copy of the web calculator. We also hope you will submit suggestions, either by emailing us or by making issues or pull requests directly on Github.

Our group house has been using this model as the basis of a shared agreement/protocol, based on a budget of 3,000 microCOVIDs per year to spend outside the house (about 58 per week). We know of another group house that (last we heard) was operating on *4* microCOVIDs per week!

We hope this helps you personally live a better pandemic life with more safety and more flexibility.

(also linkposted to the EA Forum)
Pong from pixels without reading "Pong from Pixels" 
by Ian McKenzie — 2020-08-29T17:26:25.656Z

At the beginning of this summer I finished an undergraduate degree in maths and physics and I decided to spend some time preparing for my master’s degree in AI by learning some reinforcement learning (RL). It’s not like there was a whole lot else to do this summer anyway. Before going into this I had done a module on applied ML (which basically consisted of putting data into scikit-learn functions without looking deeply at how they worked) and a general idea of the basic structure of neural networks.

The first part of the post will outline the steps I took in learning ML and RL in case anyone with a similar background is interested, and in the second part I will discuss the challenges of implementing a Deep Q-Network (DQN) algorithm on Pong directly from the original paper. I’ll also compare my approach and experience to the blog post Deep Reinforcement Learning: Pong from Pixels by Andrej Karpathy, which I didn't read until after I'd written my DQN implementation.

Yes, this game was heavily cherry-picked but at least it works some of the time!

Part I - Background

I started by looking at Spinning Up by OpenAI and reading their introduction. While reading it I thought I was understanding it fairly well but when it came time to try the exercises and implement algorithms for myself I realised I had no clue what was going on and decided to take a step back.

I’m always happiest when I understand a topic from the very basics. It’s why I preferred pure maths to applied - you start from the axioms and rarely have to rely on anything that you haven’t proved yourself earlier. For this reason I started reading Sutton and Barto’s Reinforcement Learning: An Introduction (RLAI), a textbook by two of the early big names in RL. I haven’t read any other RL textbooks but I thoroughly enjoyed the style and pacing of this book - plenty of explanation and exercises.

I read RLAI until I reached a section on neural networks (Chapter 9), at which point I switched to Goodfellow’s Deep Learning (DL), a modern classic on (deep) neural networks. I found this book a little harder to follow as there were no exercises and fewer simple examples, but it was very helpful nonetheless.

To ensure that I was understanding, I wrote my own symbol-to-symbol backpropagation code in Python and used it to construct a basic neural net capable of achieving ~90% accuracy on MNIST. I also implemented a couple...
Meaningful Rest
by Neel Nanda — 2020-08-29T15:50:05.782Z
Link: https://www.neelnanda.io/blog/29-meaningful-rest

An exercise: Set a 5 minute timer, and list the things you want to do when you feel tired and low-energy. Then, set another 5 minute timer, and list the things you feel rejuvenated after having done - the things you like doing when low-energy.

If you’re anything like me, these lists are basically disjoint! When I’m tired, I want to compulsively check things - Facebook, email, the news. I want to procrastinate: to compulsively scroll through Reddit, trashy web-fiction, the latest webcomics. But, empirically, after doing this I don’t feel any happier or more energised. Often, I feel even more tired! While the things I feel happy to have done tend to be completely different: going for a walk, meditating, reading a book. Generally, things that involve going outside and getting away from screens. Yet, these are not the things I reflexively reach for when tired. This is both a puzzle (which seems to have a solid neuroscientific basis) and obviously terrible.

This is terrible, because the things I want to do are the default actions - the things that take no activation energy to start, the things I reflexively reach for. And when I’m tired, I lack the willpower to do anything more ambitious, and will just reach for whatever is most available. And this creates a feedback loop - I am tired, so I do the things I want to do, I am not rejuvenated and made more tired, etc. This both consumes a lot of time, and doesn’t even make me happy in the process!

So, this is a problem. And fixing this is a big deal, because being well-rested and high-energy is super important. It’s key to my productivity - when I’m tired, I find it easy to procrastination, fail to make progress, and go in circles of wasted motion. And it’s bad for my happiness - being tired makes me irritated, fatigued, insecure, etc. I’ve noticed significant increases in my overall happiness after making progress on this problem, and related problems like my sleep.

Worse, this combines really badly with my default working style. I have a strong neurotic desire to finish things, and to fixate on my total output rather than time spent working. I’ll often push myself to complete my current task, going well beyond my allocated working time, and not being willing to take a break until I’m done. And in the process, I’ll push myself past the ...