trentmkelly/LessWrong-43k huggingface dataset:
GPTs are Predictors, not Imitators
by Eliezer Yudkowsky — 2023-04-08T19:59:13.601Z

(Related text posted to Twitter; this version is edited and has a more advanced final section.)

Imagine yourself in a box, trying to predict the next word - assign as much probability mass to the next token as possible - for all the text on the Internet.

Koan: Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text? What factors make that task easier, or harder? (If you don't have an answer, maybe take a minute to generate one, or alternatively, try to predict what I'll say next; if you do have an answer, take a moment to review it inside your mind, or maybe say the words out loud.)

Consider that somewhere on the internet is probably a list of thruples: <product of 2 prime numbers, first prime, second prime>.

GPT obviously isn't going to predict that successfully for significantly-sized primes, but it illustrates the basic point:

There is no law saying that a predictor only needs to be as intelligent as the generator, in order to predict the generator's next token.

Indeed, in general, you've got to be more intelligent to predict particular X, than to generate realistic X. GPTs are being trained to a much harder task than GANs.

Same spirit: <Hash, plaintext> pairs, which you can't predict without cracking the hash algorithm, but which you could far more easily generate typical instances of if you were trying to pass a GAN's discriminator about it (assuming a discriminator that had learned to compute hash functions).

Consider that some of the text on the Internet isn't humans casually chatting. It's the results section of a science paper. It's news stories that say what happened on a particular day, where maybe no human would be smart enough to predict the next thing that happened in the news story in advance of it happening.

As Ilya Sutskever compactly put it, to learn to predict text, is to learn to predict the causal processes of which the text is a shadow.

Lots of what's shadowed on the Internet has a *complicated* causal process generating it.

Consider that sometimes human beings, in the course of talking, make errors.

GPTs are not being trained to imitate human error. They're being trained to *predict* human error.

Consider the asymmetry between you, who makes an error, and an outside mind that knows you well enough and in enough detail to predic...
4 generations of alignment
by qbolec — 2023-04-08T19:59:04.199Z

Today, when I was scrapping a sticker urging to "stop immigration!" from a bus station, my son asked me what was the sticker's message about, and I've said it was calling for hate, and he asked "What is 'hate'?". My wife had tears of joy in eyes, and for once I had this feeling that perhaps I didn't do everything wrong as a father: my 8 y.o. didn't have to learn yet what is hate.

This made me realize that this process was multi-generational.

0. The generation of my grand-parents participated in WW2, and had to witness and adapt to a lot of cruelty and wild behavior. Perhaps that's why they rarely talk about their childhood. Yet the books say clearly that everyone was capable of wrongdoing, and OTOH many were capable of heroic acts, too.

1. The generation of my parents knew that the world can be a nasty place, but made conscious effort to focus on good parts, and role model good behavior. They've sometimes slipped in their language, or uncovered some dirty parts of humanity, but generally tried to make my childhood and character good.

2. My generation, knew what was the goal by observing our parents. We still saw some acts of badness, but rarely from our parents - we knew we are to mimic the parents, not the bad guys. We've raised our kids on perhaps a bit too censored and too politically correct books, cartoons, and games. We've tried ourselves to be good children according to our parents expectations and role models. "Fake it till you make it" makes it difficult to judge from the inside if we are just good at faking or become what we pursued, and I think it's a rabbit hole to try to properly define the difference (and that's why we need stage 3..)

3. My kids, were raised in a society in which most of the adults played the good guys. Unfriendly interactions are hidden from their sight, perhaps in late night news, or on stickers scrapped by parents. The goal is not, so much to make them unable to think about these things ever, but rather give them a saner starting point for such discussions and reference point they can compare proposed policies to.

I was thinking how much this is analogous to following plan for alignment:

0. The internet is a place where you have a mixed bag of good and bad people writing content which good and bad on many axes.

1. You can train a GPT-4 to be able to model all of that - an alien actress capable of role playing any part of that. You c...
The surprising parameter efficiency of vision models
by beren — 2023-04-08T19:44:36.186Z

Crossposted from my personal blog.

Epistemic status: This is a short post meant to highlight something I do not yet understand and therefore a potential issue with my models. I would also be interested to hear if anybody else has a good model of this.

Why do vision (and audio) models work so well despite being so small? State of the art models like stable diffusion and midjourney work exceptionally well, generating near-photorealistic art and images and give users a fair degree of controllability over their generations. I would estimate with a fair degree of confidence that the capabilities of these models probably surpass the mental imagery abilities of almost all humans (they definitely surpass mine and a number of people I have talked to). However, these models are also super small in terms of parameters. The original stable diffusion is only 890M parameters.

In terms of dataset size, image models are at a rough equality with humans. The stable diffusion dataset is 2 billion images. Assuming that you see 10 images per second every second you are awake and that you are awake 18 hours a day, you can observe 230 million images per year and so get the same data input as stable diffusion after 10 years. Of course, the images you see are much more redundant and we made some highly aggressive assumptions but after a human lifetime being in the same OOM as a SOTA image model is not insane. On the other hand, the hundreds of billions to trillions of tokens fed to LLMs is orders of magnitude beyond what humans could ever experience.

A similar surprising smallness occurs in audio models. OpenAI's Whisper can do almost flawless audio transcription (including multilingual translation!) with just 1.6B parameters.

Let's contrast this to the brain. Previously, I estimated that we should expect the visual cortex to have on the order of 100B parameters, if not more. The auditory cortex should be of roughly the same order of magnitude, but slightly smaller than the visual cortex. That is two orders of magnitude larger than state of the art DL models in these modalities.

This contrasts with state of the art language models which appear to be approximately equal to the brain in parameter count and abilities. Small (1-10B) language models are clearly inferior to the brain at producing valid text and completions as well as standard question answering and factual recall tasks. Human p...
Can we evaluate the "tool versus agent" AGI prediction?
by Xodarap — 2023-04-08T18:40:29.316Z

In 2012, Holden Karnofsky[1] critiqued MIRI (then SI) by saying "SI appears to neglect the potentially important distinction between 'tool' and 'agent' AI." He particularly claimed:

Is a tool-AGI possible? I believe that it is, and furthermore that it ought to be our default picture of how AGI will work

I understand this to be the first introduction of the "tool versus agent" ontology, and it is a helpful (relatively) concrete prediction. Eliezer replied here, making the following summarized points (among others):

Tool AI is nontrivial
Tool AI is not obviously the way AGI should or will be developed

Gwern more directly replied by saying:

AIs limited to pure computation (Tool AIs) supporting humans, will be less intelligent, efficient, and economically valuable than more autonomous reinforcement-learning AIs (Agent AIs) who act on their own and meta-learn, because all problems are reinforcement-learning problems.

11 years later, can we evaluate the accuracy of these predictions?

^

Some Bayes points go to LW commenter shminux for saying that this Holden kid seems like he's going places
Relative Abstracted Agency
by Audere — 2023-04-08T16:57:38.556Z

Note: This post was pasted without much editing or work put into formatting. I may come back and make it more presentable at a later date, but the concepts should still hold.

Relative abstracted agency is a framework for considering the extent to which a modeler models a target as an agent, what factors lead a modeler to model a target as an agent, and what sort of models have the nature of being agent-models. The relative abstracted agency of a target relative to a reasonably efficient modeler is based on the most effective strategies that the modeler uses to model the target, which exist on a spectrum from terminalizing strategies to simulating strategies.

Terminalizing

Relies most heavily on models of the target’s goals or utility function, and weighs future outcomes heavily based on how they might score in the target’s utility function. The modeler might ask, “what outcomes rank highly in the target’s utility function?” and use its approximations of the answer to predict future outcomes without predicting much in the way of particular actions, strategies, or paths that the target might take, except possibly to come up with lower bounds on rankings of outcomes in the target’s preferences.
Examples: the efficient market hypothesis, a savvy amateur predicting the outcome of a chess game against Stockfish or Magnus Carlsen, humans predicting what the world might look like 1 year after a superintelligent paperclip maximizer is created

Projecting

Combines models of the target’s goals or utility function with the modeler’s ability to find actions or strategies to achieve goals. The modeler might ask, “what would be the best actions or strategies to take if I had the target’s goals and resources?” and uses its approximations of the answer to predict and model the target.
Examples: a competent chess player playing chess against a level of stockfish that they have a small but non-negligible chance of defeating, large portions of game theory and decision theory, AlphaChess training itself using self-play

Psychologizing

Combines models of more specific aspects of the target’s processes, tendencies, flaws, weaknesses, or strengths with the modeler’s ability to find actions or strategies to achieve goals. The modeler might ask, “what would be the best actions or strategies to take if I had the target’s goals or motivations?”, and also ask “in what ways are the target’s s

...
The benevolence of the butcher
by dr_s — 2023-04-08T16:29:04.589Z

A few days ago I published this post on the risks of powerful transformative AGI (by which I meant AGI that takes off fast and pretty much rules the world in no time), even if aligned. Among the comments there was one by Paul Christiano which I think was very interesting, but also focused on a different scenario, one of slower take off in which AGI stays with us as a regular part of our economy for a bit longer. This post is an elaboration of the answer I gave there, because it brought to my mind a different kind of risk.

It's a common in rebuttals to pessimism about AGI to compare it to other past technologies, and how eventually they all ended up boosting productivity and thus raising the average human welfare in the long run (though I would also suggest that we do not completely ignore the short run: after all, we're most likely to live through it. I don't just want the destination to be nice, I want the trip to be reasonably safe!). I worry however that carrying this way of thinking to AGI might be a case of a critical error of extrapolation - applying knowledge that worked in a certain domain to a different domain in which some very critical assumptions on which that knowledge relied aren't true any more.

Specifically, when one thinks of any technology developed during or after the industrial revolution, one thinks of a capitalist, free-market economy. In such an economy, there are people who mostly own the capital (the land, the factories, and any other productive infrastructure) and there are people who mostly work for the former, putting the capital to use so it can actually produce wealth. The capital acts as a force multiplier which makes the labour of a single human be worth tens, hundreds, thousands of times what it would have been in a pre-industrial era; but ultimately, it is still a multiplier. A thousand times zero is zero: the worker is still an essential ingredient. The question of how this surplus in productivity is to be split fairly between interests that reward the owner of the capital for the risk they take and salary for the worker who actually put in the labour has been a... somewhat contentious issue throughout the last two centuries, but all capitalist economies exist in some kind of equilibrium that is satisfactory enough to at least not let the social fabric straight up unravel itself. Mostly, both groups need each other, and not j...
SERI MATS - Summer 2023 Cohort
by Aris — 2023-04-08T15:32:56.737Z

Applications have opened for the Summer 2023 Cohort of the SERI ML Alignment Theory Scholars Program! Our mentors include Alex Turner, Dan Hendrycks, Daniel Kokotajlo, Ethan Perez, Evan Hubinger, Janus, Jeffrey Ladish, Jesse Clifton, John Wentworth, Lee Sharkey, Neel Nanda, Nicholas Kees Dupuis, Owain Evans, Victoria Krakovna, and Vivek Hebbar.

Applications are due on May 7, 11:59 pm PT. We encourage prospective applicants to fill out our interest form (~1 minute) to receive program updates and application deadline reminders! You can also recommend that someone apply to MATS, and we will reach out and share our application with them.

Program details

SERI MATS is an educational seminar and independent research program that aims to provide talented scholars with talks, workshops, and research mentorship in the field of AI alignment, and connect them with the Berkeley alignment research community. Additionally, MATS provides scholars with housing and travel, a co-working space, and a community of peers. The main goal of MATS is to help scholars develop as alignment researchers.

Timeline

Based on individual circumstances, we may be willing to alter the time commitment of the scholars program and allow scholars to leave or start early. Please tell us your availability when applying. Our tentative timeline for applications and the MATS Summer 2023 program is below.

Pre-program

Application release: Apr 8
Application due date: May 7, 11:59 PT
Acceptance status: Mid-Late May

Summer program

Program Start: Early Jun. The Summer 2023 Cohort consists of three phases.
Training phase: Early-Late Jun
Research phase: Jul 3-Aug 31
Extension phase: ~ Sep-Dec

Training phase

A 4-week online training program (10 h/week for two weeks, then 40 h/week for two weeks). Scholars receive a stipend for completing this part of MATS (historically, $6k).

Scholars whose applications are accepted join the training phase. Mentors work on various alignment research agendas — when a mentor accepts a scholar, the scholar is considered to have entered that mentor’s specific “research stream.” At the end of the training phase, scholars will, by default, transition into the research phase.

MATS has a strong emphasis on education in addition to fostering independent research. The training program hosts an advanced alignment research curriculum, mentor-specific reading lists, workshops on model-building and ration...
AI Proposals at ‘Two Sessions’: AGI as ‘Two Bombs, One Satellite’?
by Derek M. Jones — 2023-04-08T11:31:20.655Z
Link: https://www.chinatalk.media/p/ai-proposals-at-two-sessions-agi

...

On March 4, the opening ceremony of the 14th National Committee of the Chinese People’s Political Consultative Conference (CPPCC) was held. Zhu Songchun, a member of the CPPCC and director of the Beijing Institute for General Artificial Intelligence, suggested in a proposal that China should elevate the development of AGI to the level of the “Two Bombs, One Satellite” of the contemporary era and seize the global strategic high ground of technology and industrial development.

...
All images from the WaitButWhy sequence on AI
by trevor — 2023-04-08T07:36:06.044Z

Lots of people seem to like visual learning. I don't see much of an issue with that. People who have fun with thinking tend to get more bang for their buck.

It seems reasonable to think that that Janus's image of neural network shoggoths makes it substantially easier for a lot of people to fully operationalize the concept that RLHF could steer humanity off of a cliff:

Lots of people I've met said that they were really glad that they encountered Tim Urban's WaitButWhy blog post on AI back in 2015, which was largely just a really good distillation of Nick Bostrom's Superintelligence (2014). It's a rather long (but well-written) post, so what impressed me was not the distillation, but the images.

The images in the post were very vivid, especially in the middle. It seems to me like images can work as a significant thought aid, by leaning on visual memory to aid recall, and/or to make core concepts more cognitively available during the thought process in general.

But also, almost by themselves, the images do a pretty great job describing the core concepts of AI risk, as well as the general gist of the entirety of Tim Urban's sequence. Considering that he managed to get that result, even though the post itself is >22,000 words (around as long as the entire CFAR handbook), maybe Tim Urban was simultaneously doing something very wrong and very right with writing the distillation; could he have turned a 2-hour post into a 2-minute post by just doubling the number of images?

If there was a true-optimal blog post to explain AI safety for the first time, to an otherwise-uninterested layperson (a very serious matter in AI governance), it wouldn't be surprising to me if that true-optimal blog post contained a lot of images. Walls of text are inevitable at some point or another, but there's the old saying that a picture's worth a thousand words. Under current circumstances, it makes sense for critical AI safety concepts to be easier and less burdensome to think and learn about for the first time, rather than harder and more burdensome to think about for the first time.

I've found the above two pictures particularly helpful, for doing an excellent job depicting the scope. Out of all the images that could be used to help describe AGI to someone for the first time, I would pick those two.

These two images are pretty good as well, as a primer for the curren...
Guidelines for productive discussions
by ambigram — 2023-04-08T06:00:07.363Z

I enjoyed reading Duncan_Sabien's post, Basics of Rationalist Discourse, as well as the response posts and comments. The guidelines probably wouldn't have made sense to my beginner-self though, so I wanted to try creating a list that is more targeted at beginners. In that sense, this is more of a complement to the other pieces than an alternate take.

I am not really sure if I've captured everything important, and I don't really like the "Other notes" section because it feels randomly put together. What I'm most curious about is whether taking this approach can help address some of the criticisms of the original post. I've added some of my thinking behind this piece to the appendix.

On a site like Less Wrong, we want to promote discussions and conversations that help members improve at rationality. We want participants and the audience to walk away from discussions with a clearer, more accurate understanding of the world, one which enables them to act more effectively.

Here are some guidelines for more fruitful conversations:

1) Seek to be Less Wrong instead of More Right

It feels good to be right, to show that we are correct and others are wrong. However, this can be counter-productive. It encourages us to find ways to prove we are right instead of figuring out whether or not our beliefs are right in the first place. This makes it much less likely for us to arrive at the truth.

Let's instead focus on how we can be Less Wrong:

When you hear something that doesn't make sense, get curious instead of dismissive. After all, we each experience a very small slice of reality. If someone has a seemingly nonsensical opinion, maybe it's because we're missing out a perspective.
Steelman: When someone presents a criticism that you believe is flawed, see if you can improve the criticism and use it to identify a gap in your thinking.
Ask yourself, if you were wrong, how would you find out? If nothing can change your mind, further discussion would be pointless.

2) First, check your understanding

Communication is hard. Words mean different things to different people, and we add our own interpretations to things we hear. Make sure you are addressing what the other person is saying, and not just talking past each other. Sometimes that can mean having to do some background reading.

Paraphrase to check if you've understood the points. If you think that there is something being i

...
All AGI Safety questions welcome (especially basic ones) [April 2023]
by steven0461 — 2023-04-08T04:21:36.258Z

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

AISafety.info - Interactive FAQ

Additionally, this will serve as a way to spread the project Rob Miles' team[1] has been working on: Stampy and his professional-looking face aisafety.info. This will provide a single point of access into AI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that!

Stampy - Here to help everyone learn about stamp maximization AGI Safety!

You can help by adding questions (type your question and click "I'm asking something else") or by editing questions and answers. We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase and volunteer developers to help with the conversational agent and front end that we're building.

We've got more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question.
Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!).
Also feel free to ask questions that you'r

...