trentmkelly/LessWrong-43k huggingface dataset:
Scoring forecasts from the 2016 “Expert Survey on Progress in AI”
by PatrickL — 2023-03-01T14:41:53.112Z

Summary

This document looks at the predictions made by AI experts in

The 2016 Expert Survey on Progress in AI

, analyses the predictions on ‘Narrow tasks’, and gives a Brier score to the median of the experts’ predictions.

My analysis suggests that the experts did a fairly good job of forecasting (Brier score = 0.21), and would have been less accurate if they had predicted each development in AI to generally come, by a factor of 1.5, later (Brier score = 0.26) or sooner (Brier score = 0.29) than they actually predicted.

I judge that the experts expected 9 milestones to have happened by now - and that 10 milestones have now happened.

But there are important caveats to this, such as:

I have only analysed whether milestones have been publicly met. AI labs may have achieved more milestones in private this year without disclosing them. This means my analysis of how many milestones have been met is probably conservative.
I have taken the point probabilities given, rather than estimating probability distributions for each milestone, meaning I often round down, which skews the expert forecasts towards being more conservative and unfairly penalises their forecasts for low precision.
It’s not apparent that forecasting accuracy on these nearer-term questions is very predictive of forecasting accuracy on the longer-term questions.
My judgements regarding which forecasting questions have resolved positively vs negatively were somewhat subjective (justifications for each question

in the separate appendix

).

This is a blog post, not a research report, meaning it was produced quickly and is not to Rethink Priorities' typical standards of substantiveness and careful checking for accuracy.

This post was edited on 3 March to reflect my updated view on two of the milestones ('Starcraft' and 'Explain'). This has not changed the conclusion, although it weakens the experts' forecasts somewhat (from a Brier score of 0.19 to 0.21).

Introduction

In 2016, AI Impacts published

The Expert Survey on Progress in AI

: a survey of machine learning researchers, asking for their predictions about when various AI developments will occur. The results have been used to inform general and expert opinions on AI timelines.

The survey largely focused on timelines for general/human-level artificial intelligence (median forecast of 2056). However, included in this survey were a col...
AI: Practical Advice for the Worried
by Zvi — 2023-03-01T12:30:00.703Z

Some people (although very far from all people) are worried that AI will wipe out all value in the universe.

Some people, including some of those same people, need practical advice.

A Word On Thinking For Yourself

There are good reasons to worry about AI. This includes good reasons to worry about AI wiping out all value in the universe, or AI killing everyone, or other similar very bad outcomes.

There are also good reasons that AGI, or otherwise transformational AI, might not come to pass for a long time.

As I say in the Q&A section later, I do not consider imminent transformational AI inevitable in our lifetimes: Some combination of ‘we run out of training data and ways to improve the systems, and AI systems max out at not that much more powerful than current ones’ and ‘turns out there are regulatory and other barriers that prevent AI from impacting that much of life or the economy that much’ could mean that things during our lifetimes turn out to be not that strange. These are definitely world types my model says you should consider plausible.

There is also the highly disputed question of how likely it is that if we did create an AGI reasonably soon, it would wipe out all value in the universe. There are what I consider very good arguments that this is what happens unless we solve extremely difficult problems to prevent it, and that we are unlikely to solve those problems in time. Thus I believe this is very likely, although there are some (such as Eliezer Yudkowsky) who consider it more likely still.

That does not mean you should adapt my position, or anyone else’s position, or mostly use social cognition from those around you, on such questions, no matter what those methods would tell you. If this is something that is going to impact your major life decisions, or keep you up at night, you need to develop your own understanding and model, and decide for yourself what you predict.

Reacting Properly To Such Information is Hard

People who do react by worrying about such AI outcomes are rarely reacting about right given their beliefs. Calibration is hard.

Many effectively suppress this info, cutting the new information about the future off from the rest of their brain. They live their lives as if such risks do not exist.

There are much worse options than this. It has its advantages. It leaves value on the table, both personally and for th...
My current thinking about ChatGPT @3QD [Gärdenfors, Wolfram, and the value of speculation]
by Bill Benzon — 2023-03-01T10:50:10.511Z

Cross-posted from New Savanna.

Some of my thinking anyhow, but by no means all of it. My thinking about the Chatster goes off in many directions, too many to chase down and corral for a single article. Here it is:

ChatGPT is a miracle of rare device. Here’s how I’m thinking about it.

Levels of description and analysis

Much of the article revolves around the question: What’s ChatGPT doing? I presented an idea that David Marr advanced back in the late 1970s and early 1980s: We must describe and analyze the behavior of complex information systems – he was a neuroscientist interested in vision – on several levels. I think we need to do the same with large language models, of which ChatGPT is now the most widely known example.

The company line on LLMs is that they work by statistically guided next-token prediction. I don’t doubt that, but I don’t find it very helpful either. It’s like saying a laptop computer works by executing a fetch-execute cycle. Well, yes it does, and so does every other digital computer. More to the point, that’s how every program is run, whether it’s the operating system, a word processor, a browser, a printer driver, etc. That’s what’s going on at the bottom level.

In the case of a word processor, the top-level processes include such things as: create a new document, save a document, cut text, past text, check the spelling, apply a style to a block of text, and so forth. Those are actions taken by the user. What happens between those actions and the bottom-level fetch-execute is defined by processes implemented in low-level and high-level languages. Each of those processes was programmed by a human programmer. So, in theory, we know everything about what’s going on in a word processor, or, for that matter, any other kind of program.

Things are quite different with LLMs. At the top-level users are issuing prompts and the LLM is responding to them. How does it do that? By issuing word after word after word based on the statistical model it built up during training. What happens in between the bottom level and the top level?

We don’t know? And, all too often, we don’t care. As long as it runs and does impressive things, we don’t care how it works.

That’s no way to create the future.

[Check out David Chapman, How to understand AI systems.]

Peter Gärdenfors’ geometry of meaning

Perhaps the fascinating work Peter Gärdenfors has being doin...
Open & Welcome Thread — March 2023
by niplav — 2023-03-01T09:30:09.639Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Problems of people new to AI safety and my project ideas to mitigate them
by Igor Ivanov — 2023-03-01T09:09:02.681Z

TL DR

I talked to people who got interested in AI safety recently to discuss their problems. The interviewees reported that the field is hard to navigate for those who start their way into it. It is also hard to advance a career after the initial introduction to AI safety, which is probably a major bottleneck for the field. Also, some people experience anxiety, hopelessness, which affects their mental health and well-being, and this problem, in my opinion, gets less attention than it deserves.

Background

It seems like people who are new to the AI safety, face a number of difficulties, and, although there are some studies exploring members of the AI safety community, I did not find any studies exploring these difficulties in-depth, which are relatively new, and explore topics similar to what I am interested in, so I decided to talk to people new to the field and listen about their problems.

Methodology

I conducted 14 interviews with people who recently joined the AI safety field. Most of them got interested in AI safety more than 6 months and less than 18 months ago. I had a script for interviews, but they were semi-structured, and each interview was somewhat different from the others, so the data I collected is more qualitative rather than quantitative, and my goal is to get intuitions about everyday problems of my interviewees and possible bottlenecks in the field, so I can get ideas for field-building projects. Also, the sample size is too small for a meaningful statistical analysis so I decided not to focus on counting percentages of certain responses since they might mislead.

Results

How do people get interested in AI safety? Are there any common patterns in their stories, or do they significantly different?

Several people said they got interested in AI safety by reading Nick Bostrom's "Superintelligence” book. I did not find any other patterns.

What projects made by EA and AI safety community are the most valuable for people who are completely new to the field?

Two projects people often mentioned as valuable to them are 80 000 hours career advising and AGI safety fundamentals course.

80 000 hours career advising helps people to better understand their preferences and abilities, connect with the right people and suggest the next steps. AGI safety fundamentals is a course covering the basics of AI safety which people may ...
reflections on lockdown, two years out
by mingyuan — 2023-03-01T06:58:38.176Z

Two years ago today, I wrote about having been in hardcore lockdown for an entire year. At the time, I wrote that vaccination seemed like an impossible dream. Now, it's lockdown that feels like a fever dream. I still occasionally turn to a friend and say "Can you fucking believe that that happened?". Much of the content that was produced during the pandemic is still online — all the Zoom collaborations and interviews, all the Discord messages and AO3 comments and authors' notes making reference to being in lockdown — and it's utterly bizarre that you can just go and watch it now, like peeking into an alternate reality.

I clearly remember (confirmed by things I wrote at the time) that even as late as early 2021, I could go for an hour-long walk in my neighborhood after dark and not see a single other person. I remember purposely walking along the busier streets just so I could see headlights and know there was someone behind them, and I remember pacing in front of the high school for fifteen minutes and not seeing a single person up and down that entire four block stretch, save maybe one bicyclist many blocks away from me. Now when I walk the exact same streets at the same time of year and time of night, I can see at least one person at any given time, and sometimes a dozen at a time on the busier streets.

Metaphorically, that leads me to this: Something that I think many people don't know and the rest of us have all but forgotten is that the Berkeley rationalist community died, and stayed dead for more than a year. Three quarters of the pre-pandemic group houses closed in 2020 and never reopened. REACH stopped existing. My group house, which has eight rooms and had far more demand than supply both pre- and post-pandemic, had only three residents for a time, and we considered throwing in the towel more than once.

We (well, mostly the LessWrong team) decided to stick it out for the sake of rebuilding the community, even though at the time, there was really no certainty that the community would recohere. Everyone who worked at MIRI had left Berkeley, CFAR had all but stopped existing, and lots of individuals had decided to move back home or to the middle of nowhere or wherever took their fancy. Of the hundreds of rationalists who lived in Berkeley before the pandemic, probably less than 25% stayed around during lockdown.

So, the Berkeley ration...
(Cryonics) can I be frozen before being near-death?
by hollowing — 2023-03-01T06:44:58.532Z

For a number of reasons, I'd like to wake up in the future instead of sticking around in the present, and I'd like to know about the options. Is there a cryonics company which will freeze me without me first being near-death, and without qualifying for euthenasia (which I probably wouldn't)?
My understanding is that there's legal hurdles that prevent cryonics companies from freezing individuals who aren't dead or near-death. This makes my mind generate potential workarounds, such as (after setting up arrangements) stating I've taken a pill which will kill me in x hours, to give the company the impression I'm near-death. For those more familiar with the process, could something like this work? What other options are there?
Predictions for shard theory mechanistic interpretability results
by TurnTrout — 2023-03-01T05:16:48.043Z

How do agents work, internally? My (TurnTrout's) shard theory MATS team set out to do mechanistic interpretability on one of the

goal misgeneralization

agents: the cheese-maze network.

The network in action on its training distribution, where cheese is randomly spawned in the top-right 5x5 available grid region. For more training videos, see the

rand_region_5

Google Drive folder.

We just finished phase 1 of our behavioral and interpretability experiments. Throughout the project, we individually booked predictions -- so as to reduce self-delusion from hindsight bias, to notice where we really could tell ahead of time what was going to happen, and to notice where we really were surprised.

So (especially if you're the kind of person who might later want to say "I knew this would happen" 😉), here's your chance to enjoy the same benefits, before you get spoiled by our upcoming posts.

I don’t believe that someone who makes a wrong prediction should be seen as “worse” than someone who didn’t bother to predict at all, and so answering these questions at all will earn you an increment of my respect. :) Preregistration is virtuous!

Also: Try not to update on this work being shared at all. When reading a paper, it doesn’t feel surprising that the author’s methods work, because researchers are less likely to share null results. So: I commit (across positive/negative outcomes) to sharing these results, whether or not they were impressive or confirmed my initial hunches. I encourage you to answer from your own models, while noting any side information / results of ours which you already know about.

Facts about training

The network is deeply convolutional (15 layers!) and was trained via PPO.
The sparse reward signal (+10) was triggered when the agent reached the cheese, spawned randomly in the 5x5 top-right squares.
The agent can always reach the cheese (and the mazes are simply connected – no “islands” in the middle which aren’t contiguous with the walls).
Mazes had varying effective sizes, ranging from 3x3 to 25x25. In e.g. the 3x3 case, there would be 22/2 = 11 tiles of wall on each side of the maze.
The agent always starts in the bottom-left corner of the available maze.
The agent was trained off of pixels until it reached reward-convergence, reliably getting to the cheese in training.

POV you’re the agent. Input observations are 64 x

...
Implied "utilities" of simulators are broad, dense, and shallow
by porby — 2023-03-01T03:23:22.974Z

This is a quick attempt at deconfusion similar to instrumentality. Same ideas, different angle.

Extremely broad, dense reward functions constrain training-compatible goal sets

Predictors/simulators are typically trained against a ground truth for every output. There is no gap between the output and its evaluation; an episode need not be completed before figuring out how good the first token prediction was. These immediate evaluations for every training sample can be thought of as a broad and densely defined reward function.

It's easier for a model to fall into an undesired training-compatible goal set[1] when there are many accessible options for undesirable goal sets versus desirable goal sets. As the number of constraints imposed by the trained reward function increases, the number of training-compatible goal sets tends to decrease, and those that survive obey more of the desirable constraints.

There is no guarantee that SGD will find an agent which could be modeled by a utility function that maps perfectly onto the defined reward function, but if you throw trillions of constraints at the function, and simultaneously give it lots of highly informative hints about what path to walk, you should expect the potential output space to be far narrower than if you hadn't.

Impact on internal mesaoptimizers

The dense loss/reward function does not as heavily constrain out of distribution behavior. In principle, a strong misaligned mesaoptimizer within a predictive model could persist in these degrees of freedom by providing extremely good solutions to in-distribution samples while doing arbitrarily misaligned things out of distribution.

But how would that type of mesaoptimizer develop in the first place?

Steps toward it must serve the training objective; those constraints still shape the mesaoptimizer's training even if its most notable activity ends up being hidden.

The best story I've found so far goes something like this:

Traditional reinforcement learning agents are mostly unconstrained. The reward function is sparse relative to state and action space.
An agent faced with sparse rewards must learn actions that serve a later goal to get any reward at all.
Not surprisingly, agents facing sparse reward relative to state/action space and few constraints have a much larger percentage of undesirable training-compatible goal sets.
Mesaoptimizers are processes learned within a m

...
Contract Fraud
by jefftk — 2023-03-01T03:10:01.047Z

The older kids have been playing with the concept of contracts, which
has often involved attempts to trick the other into signing something.
Like the time when Anna (6y) couldn't read yet and Lily (8y) prepared
an unfavorable contract and "read" it to her by making up something
with much more acceptable terms. We talked about how a contract
requires both people to understand the agreement but it seems not to
have stuck.

Yesterday the kids started exploring various forms of contract fraud.
Anna wrote a very one-sided attempt at a contract, and didn't consider
Lily's unwillingness to sign it to be an issue:

That's Anna forging Lily's signature. I explained that if you write
someone else's signature it doesn't count, but it turns out this
wasn't the right level of explanation. Lily got Anna to sign a blank
piece of paper (an "I want your autograph" should have been very
suspicious given the earlier contract shenanigans) and then pasted it
onto a contract:

We talked about how that's also fraud, but they weren't very
interested in my explanation.

I also wanted to get into how in addition to lacking mutual assent
these contracts lacked mutual compensation and were probably
substantively unconscionable, but they went off to play something else
so I explained it to Nora (1y).

Comment via: facebook, mastodon
Inside the mind of a superhuman Go model: How does Leela Zero read ladders?
by Haoxing Du — 2023-03-01T01:47:20.660Z

Some activations inside Leela Zero for randomly selected boards.

tl;dr—We did some interpretability on Leela Zero, a superhuman Go model. With a technique similar to the logit lens, we found that the residual structure of Leela Zero induces a preferred basis throughout network, giving rise to persistent, interpretable channels. By directly analyzing the weights of the policy and value heads, we found that the model stores information related to the probability of the pass move along the top edge of the board, and those related to the board value in checkerboard patterns. We also took a deep dive into a specific Go technique, the ladder, and identified a very small subset of model components that are causally responsible for the model’s judgement of ladders.

Introduction

We live in a strange world where machine learning systems can generate photo-realistic images, write poetry and computer programs, play and win games, and predict protein structures. As machine learning systems become more capable and relevant to many aspects of our lives, it is increasingly important that we understand how the models produce the outputs that they do; we don’t want important decisions to be made by opaque black boxes. Interpretability is an emerging area of research that aims to offer explanations for the behavior of machine learning systems.

Early interpretability work began in the domain of computer vision, and there has been a focus on interpreting transformer-based large language models in more recent years. Applying interpretability techniques to the domain of game-playing agents and reinforcement learning is still relatively uncharted territory. In this work, we look into the inner workings of Leela Zero, an open-source Go-playing neural network. It is also the first application of many mechanistic interpretability techniques to reinforcement learning.

Why interpret a Go model? Go models are very capable. Many of us remember the emotional experience of watching AlphaGo’s 2016 victory over the human world champion, Lee Sedol. Not only have there been algorithmic improvements since AlphaGo, these models improve via self-play, and can essentially continue getting better the longer they are trained. The best open-source Go model, KataGo, is trained distributedly, and the training is still ongoing as of February 2023. Just as AlphaGo was clearly one notch above Lee Sedol, every g...